Review main Merged

89144[BUGFIX] Execute `bulkInsert()` in chunks for larger datasets

Oliver Heins CR +2 V +2 Normal +75 −71 updated 04/16/25, 19:59

Commit message

The `Connection::bulkInsert()` method invites to pass bulk of
data to insert into the database, using database placeholders
to provide proper value handling. However, handling for more
then one specific database system is not easy and each single
system defines a different maximum placeholder limit and lead
to exceeding that limit with a lot of data rows passed in.

To mitigate this issue a maximum data rows limit is calculated
based on the columns count and the maximal placeholder limit,
with a safety margin. The result is used to build chunks out
of the passed insert data rows and splitted into multiple calls.

This mitigates "Too many parameters" database errors inserting
larger datasets to some degree, not fully solving it.

Extensions or instances using the `bulkInsert()` method still
recieving the aforementioned error should implement chunking
of the insert data and call this method mutiple times, which
should be more an edge case after this change.

Review

  • Code review CR +2 Done
    • Stefan Bürk +2
    • Georg Ringer +1
  • Verified V +2 Done
    • Stefan Bürk +2
    • Georg Ringer +1
    • core-ci +1 CI

Issues 1

Files 3

Review log 1317

  1. Oliver Heins uploaded patch set 1 04/16/25, 12:47

  2. core-ci voted on patch set 1 V +1 04/16/25, 12:55 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/77838

  3. Georg Ringer commented on patch set 1 04/16/25, 15:26

    Connection.php:211

    I suggest chunkSize

    Connection.php:254

  4. Stefan Bürk commented on patch set 1 CR -1 04/16/25, 16:37

    On it ... reviewing/testing/changing ...

    Connection.php:211 Reply

    Either chunkSize should be handed from the outside or calculated internally. As this is a change in signature, not breaking I would abstain from making this option passed and instead calculate the chunks everytime. There should be no need to change that from the outside. If that is needed, calling code can do the chunk on themself and add chunked $data directly.
  5. Stefan Bürk uploaded patch set 2 CR -1 V +1 04/16/25, 17:07

  6. core-ci voted on patch set 2 V -1 04/16/25, 17:17 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/77848

  7. Stefan Bürk uploaded patch set 3 V -1 04/16/25, 17:43

  8. Stefan Bürk commented on patch set 3 04/16/25, 17:43

    Reply

    Done

    Connection.php:211 Reply

    Removed the new argument again.

    Connection.php:254 Reply

    Ack.
  9. core-ci voted on patch set 3 V +1 04/16/25, 17:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/77853

  10. Stefan Bürk commented on patch set 3 04/16/25, 17:54

    Thanks @o.heins@leuchtfeuer.com for this fix. ❤️
  11. Stefan Bürk voted on patch set 3 CR +1 V +1 04/16/25, 17:54

  12. Stefan Bürk updated the commit message in patch set 4 CR +1 V +1 04/16/25, 18:07

  13. Stefan Bürk voted on patch set 4 CR +1 V +1 04/16/25, 18:07

  14. core-ci voted on patch set 4 V +1 04/16/25, 18:14 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/77854

  15. Georg Ringer voted on patch set 4 CR +1 V +1 04/16/25, 18:25

  16. Stefan Bürk voted on patch set 4 CR +2 V +2 04/16/25, 18:57

  17. Stefan Bürk merged patch set 5 04/16/25, 19:59