Review main Merged

62749[BUGFIX] Reduce memory consumption in findIdentifiersByTag()

Markus Klösges CR +2 V +2 Easy +9 −21 updated 01/24/20, 16:56

Commit message

When searching cache-identifiers by tag, the Typo3DatabaseBackend now
utilizes DBMS functions for de-duplication of identifiers instead
of fetching all records and de-duplicate them afterwards in php space.
This dramatically reduces memory usage, when many duplicated identifiers
are in a cache table for any given tag.
Note that there is no noticable impact whenever there are no duplicates
at all, as the DBMS handles the de-duplication.

Review

  • Code review CR +2 Done
    • Susi Moog +2
    • Daniel Goerz +1
    • Jonas Eberle +1
  • Verified V +2 Done
    • Susi Moog +2
    • Jonas Eberle +1
    • TYPO3com +1 CI

Issues 1

Files 1

Review log 2432

Show 57 earlier messages
  1. Markus Klösges uploaded patch set 1 12/19/19, 10:29

  2. TYPO3com voted on patch set 1 V -1 12/19/19, 10:36 Completed build in 07m 02s on Thu, 19 Dec, 10:36 AM Test Summary: 69116 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39527

  3. TYPO3com voted on patch set 1 V +1 12/19/19, 10:43 Completed build in 07m 09s on Thu, 19 Dec, 10:43 AM Test Summary: 69128 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39528

  4. Jonas Eberle commented on patch set 1 12/19/19, 11:36

    As to my knowledge there is no SELECT DISTINCT available in QueryBuilder, I think a groupBy() is actually the best way.

    Typo3DatabaseBackend.php:242

    I think nothing requires the keys to be set in the return array so we could get rid of this loop altogether. I suggest replacing it with $cacheEntryIdentifiers = $result->fetchAll(\Doctrine\DBAL\FetchMode::COLUMN, 0).
  5. Markus Klösges commented on patch set 1 12/19/19, 11:51

    Typo3DatabaseBackend.php:242 Reply Unresolved

    I would be totally okay with that. But I don't know anything about the innards of the Caching-Framework, so I didn't want to touch that part. But sure, it isn't documented anyways, anywhere - so lets go for that.
  6. Markus Klösges uploaded patch set 2 12/19/19, 12:03

  7. Markus Klösges commented on patch set 2 12/19/19, 12:08

    Went ahead and refactored that pattern throughout the whole class
  1. TYPO3com voted on patch set 2 V -1 12/19/19, 12:13 Completed build in 10m 37s on Thu, 19 Dec, 12:13 PM Test Summary: 3 of 69578 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39530

  2. Jonas Eberle commented on patch set 2 12/19/19, 12:14

    They use PDO:: constants in the class. I am not sure which is better. Core team might have a say.
  3. Markus Klösges commented on patch set 2 12/19/19, 12:16

    Now tests are failing. They are explicitly testing for these keys. I blamed myself through the git history. The keys are set since the beginning of this file in 2008. Don't know how to proceed here. Core -Team opinion is highly appreciated
  4. Jonas Eberle commented on patch set 2 CR +1 12/19/19, 12:16

    We will have to investigate the tests failure. To me it looks like it is just the assumption of the test that it gets array keys. Nothing functional failing.
  5. Jonas Eberle uploaded patch set 3 12/19/19, 20:49

  6. Jonas Eberle uploaded patch set 4 12/19/19, 20:53

  7. Jonas Eberle commented on patch set 4 12/19/19, 20:56

    I changed the bug title and updated the functional test. Do you have some measured memory consumption? That might help.
  8. TYPO3com voted on patch set 3 V -1 12/19/19, 20:57 Completed build in 08m 19s on Thu, 19 Dec, 08:57 PM Test Summary: 3 of 69568 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39537

  9. TYPO3com voted on patch set 4 V -1 12/19/19, 21:01 Completed build in 07m 50s on Thu, 19 Dec, 09:01 PM Test Summary: 68847 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39538

  10. TYPO3com voted on patch set 4 V +1 12/20/19, 06:39 Completed build in 9h 54m 15s on Fri, 20 Dec, 06:39 AM Test Summary: 69128 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39538

  11. Markus Klösges commented on patch set 4 12/20/19, 08:36

    > Do you have some measured memory consumption? That might help. Not really. I stumbled upon that code in an OOM situation produced by EXT:staticfilecache that (imho) wrongly created about 3 million entries with the same identifier in the corresponding table. Flushing them by tag caused 3 million rows to be fetched, pumped through php space, to be condensed down to one row. Applying this patch fixed the symptom - but in my opinion it does not hurt to apply it, no matter if this circumstance (having that many duplicate records there) should happen in production at all. As such, I have a database still lying around with all those records, and could to tests, but the patch is so straight forward, that I doubt that its worth the effort.
  12. Daniel Goerz uploaded patch set 5 01/13/20, 11:00

  13. Daniel Goerz commented on patch set 5 01/13/20, 11:03

    Typo3DatabaseBackend.php:242

    In the unlikely case that somebody depends on the array keys being set to the identifiers as well, we should preserve the final array structure of this public API by applying array_combine() in this and all other places.
  14. TYPO3com voted on patch set 5 V +1 01/13/20, 11:08 Completed build in 07m 52s on Mon, 13 Jan, 11:08 AM Test Summary: 69206 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39850

  15. Markus Klösges commented on patch set 5 01/13/20, 11:26

    Typo3DatabaseBackend.php:242 Reply

    Ack. Thats why I also hesitated to change the unit tests, as they where explicitly requiring the keys. Do we have a way to properly deprecate these keys to be able to drop this with the next major version?
  16. Daniel Goerz commented on patch set 5 01/22/20, 13:48

    > Patch Set 5: > > (1 comment)

    Typo3DatabaseBackend.php:242 Reply

    No, I dont think there is any way to do this. We could do a breaking change before 11.0 ... or just leave it like it is.
  17. Daniel Goerz commented on patch set 5 01/22/20, 13:49

    Typo3DatabaseBackend.php:242 Reply

    Could you update the patch so that the array keys are preserved?
  18. Markus Klösges uploaded patch set 6 01/22/20, 14:50

  19. Markus Klösges commented on patch set 6 01/22/20, 14:54

    Typo3DatabaseBackend.php:242 Reply

    Done
  20. Susi Moog voted on patch set 6 CR +1 01/22/20, 15:00

  21. TYPO3com voted on patch set 6 V +1 01/22/20, 15:02 Completed build in 11m 22s on Wed, 22 Jan, 03:02 PM Test Summary: 70544 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40145

  22. Jonas Eberle voted on patch set 6 V +1 CR +1 01/22/20, 15:06

  23. Daniel Goerz voted on patch set 6 CR +1 01/22/20, 15:07

  24. Susi Moog voted on patch set 6 V +2 CR +2 01/24/20, 16:56

  25. Susi Moog merged patch set 7 01/24/20, 16:56