Review main Merged

93324[BUGFIX] Chunk record processing in ReferenceIndex to prevent OOM

Mikel Wohlschlegel CR +2 V +2 Normal +49 −26 updated 08/31/26, 12:58

Commit message

ReferenceIndex::updateIndex() loaded all records of a table in a
single unbounded query (SELECT * FROM table ORDER BY uid). Both
database drivers TYPO3 supports for MySQL/MariaDB buffer the
complete result set client side before the first row is handed
back: mysqli through store_result(), pdo_mysql through the default
of PDO::MYSQL_ATTR_USE_BUFFERED_QUERY. pdo_pgsql behaves the same
way. The entire table is therefore materialized in memory when
executeQuery() is called, and that allocation counts against the
PHP memory limit.

For tables with many columns and records - for example tt_content
with ~132 columns and ~90.000 records - this exhausts even
generous memory limits and lets "referenceindex:update" die with a
fatal out of memory error. Multi site installations and large
corporate or university portals commonly reach these record
counts.

Switching to an unbuffered query is no option here, since
updateRefIndexTable() issues further queries on the same
connection while the result set is being traversed.

The single unbounded query is therefore replaced with keyset
pagination using a chunk size of 5000 records. Each chunk queries
WHERE uid > :lastUid ORDER BY uid LIMIT 5000, which is backed by
the primary key and avoids the degradation of OFFSET based
pagination. The result set is freed between chunks, capping peak
memory regardless of table size, while all existing processing
logic - workspace handling, progress listener, error reporting -
stays unchanged.

A functional test covering the chunk boundary would need a data
set larger than the chunk size, which is not reasonable for the
test suite. Such a test is left out by intention. The traversal
itself stays covered indirectly by the existing reference index
tests and by assertCleanReferenceIndex() of the DataHandler
scenario tests.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Stefan Bürk +1
  • Verified V +2 Done
    • Benni Mack +2
    • Stefan Bürk +1
    • core-ci +1 CI

Issues 1

Files 1

Review log 1015

  1. Mikel Wohlschlegel uploaded patch set 1 03/24/26, 08:35

  2. Mathias Brodala commented on patch set 1 03/24/26, 08:39

    How about using the QueryBuilderPaginator instead?
  3. core-ci voted on patch set 1 V -1 03/24/26, 08:49 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98546

  4. Mikel Wohlschlegel rebased onto patch set 2 V -1 03/24/26, 09:13

  5. core-ci voted on patch set 2 V +1 03/24/26, 09:52 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/98551

  6. Benni Mack updated the commit message in patch set 3 V +1 04/22/26, 06:44

  7. core-ci voted on patch set 3 V +1 04/22/26, 11:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/100777

  8. Benni Mack rebased onto patch set 4 V +1 08/15/26, 09:32

  9. core-ci voted on patch set 4 V +1 08/15/26, 09:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106755

  10. Stefan Bürk uploaded patch set 5 V +1 08/31/26, 11:19

  11. Stefan Bürk commented on patch set 5 08/31/26, 11:20

    Reply

    Thanks for the pointer, but `QueryBuilderPaginator` is not a good fit here and I would like to keep the keyset approach. * It paginates with `setFirstResult()` (OFFSET), which is precisely the degradation this change avoids — on a 90k-row `tt_content` the late pages make the DB walk and discard almost the whole table. * `getTotalItems()` adds a CTE-wrapped `COUNT` per table, while `updateIndex()` already counted the table for the progress listener. * `AbstractPaginator::updateInternalState()` clamps an out-of-range page back to the last page instead of returning an empty result, so it cannot express "iterate until exhausted"; driving it needs a page count frozen before the run, which silently skips records when the table changes underneath. * Its docblock scopes it to presentation-layer pagination (language overlay caveats, "item count jumps on pages") — not to a low-level index rebuild. Keyset over the primary key is deterministic, index-backed and needs no count.
  12. core-ci voted on patch set 5 V +1 08/31/26, 11:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107419

  13. Stefan Bürk voted on patch set 5 CR +1 V +1 08/31/26, 11:27

  14. Benni Mack voted on patch set 5 CR +2 V +2 08/31/26, 12:58

  15. Benni Mack merged patch set 6 08/31/26, 12:58