Review main Merged

82700[BUGFIX] Prevent memory leak when fetching a lot of database records

Sascha Nowak CR +2 V +2 Easy +5 −9 updated 01/31/24, 07:01

Commit message

When fetching a lot of records from database the memory is nearly
doubled in the fetchAll function. It seems php keeps references or
creates an array copy that could not be garbage collected. Enforcing
a manuell gc_collect_cycles does not help either.

Review

  • Code review CR +2 Done
    • Benjamin Franzke +2
    • Oliver Klee +1
    • Stefan Bürk +1
  • Verified V +2 Done
    • Benjamin Franzke +2
    • Stefan Bürk +1
    • core-ci +1 CI

Issues 1

Files 1

Review log 1014

  1. Sascha Nowak uploaded patch set 1 01/29/24, 18:34

  2. Sascha Nowak updated the commit message in patch set 2 01/29/24, 18:38

  3. Stefan Bürk updated the commit message in patch set 3 01/29/24, 18:39

  4. core-ci voted on patch set 1 V +1 01/29/24, 18:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52454

  5. core-ci voted on patch set 3 V +1 01/29/24, 18:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52457

  6. core-ci voted on patch set 2 V +1 01/29/24, 18:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52456

  7. Stefan Bürk commented on patch set 3 CR -1 01/29/24, 22:13

    temp -1 to look into this tomorrow without hurry ...
  8. Stefan Bürk updated the commit message in patch set 4 CR -1 V +1 01/30/24, 15:46

  9. Stefan Bürk commented on patch set 4 CR +1 V +1 01/30/24, 15:49

    Reply

    This is quite a interesting issue. I'd say we use this fix for now at this point. I'll keep investigating this deeper and consider to create a PHP issue for this, but using https://3v4l.org/ is quite hard to provide meaningfull tests. In local tests, this fix does not only reduce the memory usage but also helps to keep the peak usage low(er). Still - quite a good catch. But in the end we need deeper investigation if this is a generic issue and array method usage should be rethinked ( array_map(), array_walk(), array_filter(), ...) - but out of scope for this change.
  10. core-ci voted on patch set 4 V +1 01/30/24, 15:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52501

  11. Oliver Klee voted on patch set 4 CR +1 01/30/24, 18:14

  12. Benjamin Franzke commented on patch set 4 CR +1 V +1 01/31/24, 06:45

    Reply

    I don't think this is a PHP issue but actually expected. `$record` must be kept as-is (by PHP) during the entire `array_map` lifecycle and still we require new memory to be written. With Sascha's patch the existing data is effectively dereferenced and thus immediately garabge collectable. Looks sane, good catch indeed.
  13. Benjamin Franzke voted on patch set 4 CR +2 V +2 01/31/24, 06:49

  14. Benjamin Franzke merged patch set 5 01/31/24, 07:01