#109334ReferenceIndex::updateIndex() causes OOM on large tables due to unbounded SELECT *
Description
ReferenceIndex::updateIndex() loads all records of a table in a single unbounded query (SELECT * FROM {table} ORDER BY uid) and iterates over them via fetchAssociative(). While fetchAssociative() fetches one row at a time from the PHP side, PDO with MySQL/MariaDB uses buffered queries by default, meaning the entire result set is loaded into PHP memory when executeQuery() is called.
For installations with large tables, this causes fatal Out-of-Memory errors during referenceindex:update.
Reproduction
Any TYPO3 installation with a large tt_content table will trigger this when running:
typo3 referenceindex:update
Memory calculation
The tt_content table in TYPO3 v13 has a large amount of columns, including several TEXT, MEDIUMTEXT, and LONGTEXT fields (bodytext, pi_flexform, l18n_diffsource, etc.).
With realistic average data per row, each record consumes approximately ~20 KB of PHP memory as an associative array (due to PHP’s hashtable overhead per key-value pair plus string storage).
| tt_content records | PHP memory (result set only) |
|---|---|
| 10,000 | ~200 MB |
| 50,000 | ~1,000 MB (1 GB) |
| 90,000 | ~1,800 MB (1.8 GB) |
This is only the buffered query result — it does not include memory used by updateRefIndexTable() processing, the existing reference index data, TYPO3 framework overhead, or other tables processed in the same run.
The default PHP memory_limit is 256M (also TYPO3’s minimum requirement). Even with a generous 1G or 2G limit, installations with 50,000+ content elements will fail. Our production instance with ~90,000 tt_content records consistently crashes with OOM.
Multi-site TYPO3 installations, university portals, and large corporate setups commonly reach these record counts — this is not an edge case.
Solution
Replace the single unbounded query with keyset pagination
Why keyset pagination over OFFSET/LIMIT:- Uses the primary key index — no performance degradation at high offsets
- Deterministic: no skipped/duplicate records even with concurrent writes
$queryResult->free()releases memory between chunks
- Peak memory: ~5,000 rows × ~20 KB = ~100 MB (constant, regardless of table size)
- No behavioral change: workspace handling, progress listener, and error reporting remain identical
- Minimal performance overhead: a few additional indexed queries vs. one massive unbounded query
Reviews 3
Related issues 1
History 211
-
Gerrit Code Review pushed patch set 1 for main to change 93324 status
NewUnder Review <span data-processor="relativetime" data-value="2026-03-24T08:35:47+00:00" > 03/24/26, 08:35 </span> -
Gerrit Code Review pushed patch set 2 for main to change 93324 <span data-processor="relativetime" data-value="2026-03-24T09:13:48+00:00" > 03/24/26, 09:13 </span>
-
Gerrit Code Review pushed patch set 3 for main to change 93324 <span data-processor="relativetime" data-value="2026-04-22T06:44:03+00:00" > 04/22/26, 06:44 </span>
-
Gerrit Code Review pushed patch set 4 for main to change 93324 <span data-processor="relativetime" data-value="2026-08-15T09:32:52+00:00" > 08/15/26, 09:32 </span>
-
Stefan Bürk changed Related to #110572 <span data-processor="relativetime" data-value="2026-08-31T11:15:40+00:00" > 08/31/26, 11:15 </span>
-
Gerrit Code Review pushed patch set 5 for main to change 93324 <span data-processor="relativetime" data-value="2026-08-31T11:19:36+00:00" > 08/31/26, 11:19 </span>
-
Gerrit Code Review pushed patch set 1 for 13.4 to change 95491 <span data-processor="relativetime" data-value="2026-08-31T12:57:25+00:00" > 08/31/26, 12:57 </span>
-
Gerrit Code Review pushed patch set 1 for 14.3 to change 95492 <span data-processor="relativetime" data-value="2026-08-31T12:57:38+00:00" > 08/31/26, 12:57 </span>
-
Gerrit Code Review pushed patch set 2 for 14.3 to change 95492 <span data-processor="relativetime" data-value="2026-08-31T12:57:54+00:00" > 08/31/26, 12:57 </span>
-
Gerrit Code Review pushed patch set 2 for 13.4 to change 95491 <span data-processor="relativetime" data-value="2026-08-31T12:58:05+00:00" > 08/31/26, 12:58 </span>
-
Mikel Wohlschlegel commented and changed status
Under ReviewResolved done0 %100 % <span data-processor="relativetime" data-value="2026-08-31T13:00:11+00:00" > 08/31/26, 13:00 </span>
Applied in changeset commit:810b43093142732d89cf16012708f571b72209f3.