Review 13.4 Merged

89147[BUGFIX] Execute `bulkInsert()` in chunks for larger datasets

Stefan Bürk CR +2 V +2 Normal +75 −71 updated 04/16/25, 20:01

Commit message

The `Connection::bulkInsert()` method invites to pass bulk of
data to insert into the database, using database placeholders
to provide proper value handling. However, handling for more
then one specific database system is not easy and each single
system defines a different maximum placeholder limit and lead
to exceeding that limit with a lot of data rows passed in.

To mitigate this issue a maximum data rows limit is calculated
based on the columns count and the maximal placeholder limit,
with a safety margin. The result is used to build chunks out
of the passed insert data rows and splitted into multiple calls.

This mitigates "Too many parameters" database errors inserting
larger datasets to some degree, not fully solving it.

Extensions or instances using the `bulkInsert()` method still
recieving the aforementioned error should implement chunking
of the insert data and call this method mutiple times, which
should be more an edge case after this change.

Review

  • Code review CR +2 Done
    • Stefan Bürk +2
  • Verified V +2 Done
    • Stefan Bürk +2
    • core-ci +1 CI

Issues 1

Files 3

Review log 34

  1. Stefan Bürk uploaded patch set 1 04/16/25, 19:40

  2. core-ci voted on patch set 1 V +1 04/16/25, 19:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/77857

  3. Stefan Bürk voted on patch set 1 CR +2 V +2 04/16/25, 19:59

  4. Stefan Bürk merged patch set 2 04/16/25, 20:01