Review main Merged

95731[BUGFIX] Remove garbage collected rows in batches

Stefan Bürk CR +2 V +2 Hard +566 −25 updated 09/30/26, 05:47

Commit message

The table garbage collection task removes all expired rows of a table
with a single DELETE statement. On a table that has grown to millions
of rows, sys_log and sys_history being the usual candidates, that
statement is one huge transaction: it holds locks for its whole
runtime and can exceed transaction size limits, which fails the task
with "Transaction size exceed set threshold" on a Galera cluster.

The expired rows are now counted first and then removed in batches of
at most 10000 rows. Every round determines the primary key value of
the last row of the next batch with an ordered and limited select,
and removes the expired rows up to that boundary, which bounds a
single statement to the size of a batch. The round that finds less
than a full batch left removes the remainder. Both statements are
prepared once per table and re-executed per round.

The number of rounds is derived from the counted rows, with one
additional round for rows expiring while the task is running. Rows
left over after that are removed by the next run of the task.

The batches are deliberately not wrapped into a transaction: every
batch is committed on its own, so concurrent operations on the table
are not blocked for the whole runtime of the task.

A table without a primary key, or with a primary key spanning several
columns, has no boundary the batches could be limited with. Such a
table, and any table with less expired rows than the size of a batch,
is still cleaned up with a single statement.

The error of a failed cleanup now names the table that was handled
instead of the selected one, which is empty when all tables are
cleaned up at once.

Review

  • Code review CR +2 Done
    • Georg Ringer +2
    • Josef Glatz +1
    • Oliver Klee +1
    • Wouter Wolters +1
    • Ioulia Kondratovitch +1
  • Verified V +2 Done
    • Georg Ringer +2
    • Josef Glatz +1
    • Oliver Klee +1
    • Wouter Wolters +1
    • core-ci +1 CI

Issues 1

Files 6

Review log 1922

  1. Stefan Bürk uploaded patch set 1 09/08/26, 16:36

  2. core-ci voted on patch set 1 V +1 09/08/26, 16:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/108104

  3. Stefan Bürk marked it work in progress 09/08/26, 17:01

  4. Stefan Bürk uploaded patch set 2 V +1 09/08/26, 17:09

  5. core-ci voted on patch set 2 V +1 09/08/26, 17:17 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/108105

  6. Ioulia Kondratovitch voted on patch set 2 CR +1 09/09/26, 14:49

  7. Stefan Bürk marked it ready for review 09/10/26, 11:28

  8. Benni Mack commented on patch set 2 09/15/26, 10:26

    I have done multiple of such changes, and I think a RST file is not needed.
  9. Oliver Klee voted on patch set 2 CR +1 V +1 09/15/26, 11:43

  10. Georg Ringer commented on patch set 2 CR +1 V +1 09/15/26, 19:15

    fine also to remove the rst!
  11. Josef Glatz commented on patch set 2 CR +1 V +1 09/15/26, 19:26

    Successfully tested. Testing on a cluster was not possible atm.
  12. Josef Glatz commented on patch set 2 09/29/26, 13:46

    Should I remove the rst to get that patch into the core?
  13. Josef Glatz commented on patch set 2 09/29/26, 14:42

    Reply

    Done
  14. Josef Glatz commented on patch set 3 CR +1 V +1 09/29/26, 14:42

    Published edit on patch set 2.
  15. Josef Glatz voted on patch set 3 CR +1 V +1 09/29/26, 14:42

  16. Ioulia Kondratovitch voted on patch set 3 CR +1 09/29/26, 14:45

  17. core-ci voted on patch set 3 V +1 09/29/26, 14:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/109511

  18. Oliver Klee voted on patch set 3 CR +1 V +1 09/29/26, 15:04

  19. Georg Ringer voted on patch set 3 CR +1 V +1 09/29/26, 15:16

  20. Wouter Wolters voted on patch set 3 CR +1 V +1 09/29/26, 20:42

  21. Georg Ringer voted on patch set 3 CR +2 V +2 09/30/26, 05:47

  22. Georg Ringer merged patch set 4 09/30/26, 05:47