62749[BUGFIX] Reduce memory consumption in findIdentifiersByTag()
Commit message
Review
-
Code review CR +2 Done
- Susi Moog +2
- Daniel Goerz +1
- Jonas Eberle +1
-
Verified V +2 Done
- Susi Moog +2
- Jonas Eberle +1
- TYPO3com +1 CI
Issues 1
Files 1
Review log 2432
Show 57 earlier messages
-
Markus Klösges uploaded patch set 1 12/19/19, 10:29
-
TYPO3com voted on patch set 1 V -1 12/19/19, 10:36 Completed build in 07m 02s on Thu, 19 Dec, 10:36 AM Test Summary: 69116 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39527
-
TYPO3com voted on patch set 1 V +1 12/19/19, 10:43 Completed build in 07m 09s on Thu, 19 Dec, 10:43 AM Test Summary: 69128 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39528
-
Jonas Eberle commented on patch set 1 12/19/19, 11:36
As to my knowledge there is no SELECT DISTINCT available in QueryBuilder, I think a groupBy() is actually the best way.Typo3DatabaseBackend.php:242I think nothing requires the keys to be set in the return array so we could get rid of this loop altogether. I suggest replacing it with $cacheEntryIdentifiers = $result->fetchAll(\Doctrine\DBAL\FetchMode::COLUMN, 0). -
Markus Klösges commented on patch set 1 12/19/19, 11:51
Typo3DatabaseBackend.php:242Reply UnresolvedI would be totally okay with that. But I don't know anything about the innards of the Caching-Framework, so I didn't want to touch that part. But sure, it isn't documented anyways, anywhere - so lets go for that. -
Markus Klösges uploaded patch set 2 12/19/19, 12:03
-
Markus Klösges commented on patch set 2 12/19/19, 12:08
Went ahead and refactored that pattern throughout the whole class
-
TYPO3com voted on patch set 2 V -1 12/19/19, 12:13 Completed build in 10m 37s on Thu, 19 Dec, 12:13 PM Test Summary: 3 of 69578 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39530
-
Jonas Eberle commented on patch set 2 12/19/19, 12:14
They use PDO:: constants in the class. I am not sure which is better. Core team might have a say. -
Markus Klösges commented on patch set 2 12/19/19, 12:16
Now tests are failing. They are explicitly testing for these keys. I blamed myself through the git history. The keys are set since the beginning of this file in 2008. Don't know how to proceed here. Core -Team opinion is highly appreciated -
Jonas Eberle commented on patch set 2 CR +1 12/19/19, 12:16
We will have to investigate the tests failure. To me it looks like it is just the assumption of the test that it gets array keys. Nothing functional failing. -
Jonas Eberle uploaded patch set 3 12/19/19, 20:49
-
Jonas Eberle uploaded patch set 4 12/19/19, 20:53
-
Jonas Eberle commented on patch set 4 12/19/19, 20:56
I changed the bug title and updated the functional test. Do you have some measured memory consumption? That might help. -
TYPO3com voted on patch set 3 V -1 12/19/19, 20:57 Completed build in 08m 19s on Thu, 19 Dec, 08:57 PM Test Summary: 3 of 69568 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39537
-
TYPO3com voted on patch set 4 V -1 12/19/19, 21:01 Completed build in 07m 50s on Thu, 19 Dec, 09:01 PM Test Summary: 68847 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39538
-
TYPO3com voted on patch set 4 V +1 12/20/19, 06:39 Completed build in 9h 54m 15s on Fri, 20 Dec, 06:39 AM Test Summary: 69128 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39538
-
Markus Klösges commented on patch set 4 12/20/19, 08:36
> Do you have some measured memory consumption? That might help. Not really. I stumbled upon that code in an OOM situation produced by EXT:staticfilecache that (imho) wrongly created about 3 million entries with the same identifier in the corresponding table. Flushing them by tag caused 3 million rows to be fetched, pumped through php space, to be condensed down to one row. Applying this patch fixed the symptom - but in my opinion it does not hurt to apply it, no matter if this circumstance (having that many duplicate records there) should happen in production at all. As such, I have a database still lying around with all those records, and could to tests, but the patch is so straight forward, that I doubt that its worth the effort. -
Daniel Goerz uploaded patch set 5 01/13/20, 11:00
-
Daniel Goerz commented on patch set 5 01/13/20, 11:03
Typo3DatabaseBackend.php:242In the unlikely case that somebody depends on the array keys being set to the identifiers as well, we should preserve the final array structure of this public API by applying array_combine() in this and all other places. -
TYPO3com voted on patch set 5 V +1 01/13/20, 11:08 Completed build in 07m 52s on Mon, 13 Jan, 11:08 AM Test Summary: 69206 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-39850
-
Markus Klösges commented on patch set 5 01/13/20, 11:26
Typo3DatabaseBackend.php:242ReplyAck. Thats why I also hesitated to change the unit tests, as they where explicitly requiring the keys. Do we have a way to properly deprecate these keys to be able to drop this with the next major version? -
Daniel Goerz commented on patch set 5 01/22/20, 13:48
> Patch Set 5: > > (1 comment)Typo3DatabaseBackend.php:242ReplyNo, I dont think there is any way to do this. We could do a breaking change before 11.0 ... or just leave it like it is. -
Daniel Goerz commented on patch set 5 01/22/20, 13:49
Typo3DatabaseBackend.php:242ReplyCould you update the patch so that the array keys are preserved? -
Markus Klösges uploaded patch set 6 01/22/20, 14:50
-
Markus Klösges commented on patch set 6 01/22/20, 14:54
Typo3DatabaseBackend.php:242ReplyDone -
Susi Moog voted on patch set 6 CR +1 01/22/20, 15:00
-
TYPO3com voted on patch set 6 V +1 01/22/20, 15:02 Completed build in 11m 22s on Wed, 22 Jan, 03:02 PM Test Summary: 70544 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40145
-
Jonas Eberle voted on patch set 6 V +1 CR +1 01/22/20, 15:06
-
Daniel Goerz voted on patch set 6 CR +1 01/22/20, 15:07
-
Susi Moog voted on patch set 6 V +2 CR +2 01/24/20, 16:56
-
Susi Moog merged patch set 7 01/24/20, 16:56