72243[BUGFIX] Reduce memory usage in row updater
Commit message
Review
-
Code review CR 0
-
Verified V 0
Issues 1
Files 5
- typo3/sysext/install/Classes/Updates/DatabaseRowsUpdateWizard.php6+108 −57
- typo3/sysext/install/Tests/Functional/Updates/RowUpdater/Fixtures/TtContentRowChunkingImport.csvnew+7 −0
- typo3/sysext/install/Tests/Functional/Updates/RowUpdater/Fixtures/TtContentRowChunkingResult.csvnew+7 −0
- typo3/sysext/install/Tests/Functional/Updates/RowUpdater/TestCandidate/RowUpdaterContentBodytextToLower.phpnew+63 −0
- typo3/sysext/install/Tests/Functional/Updates/RowUpdater/RowUpdaterRowFetchChunkingTest.phpnew+65 −0
Review log 1724
-
Sybille Peters uploaded patch set 1 11/20/21, 19:59
-
Sybille Peters marked it work in progress 11/20/21, 19:59
-
core-ci voted on patch set 1 V +1 11/20/21, 20:13 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19592
-
Sybille Peters uploaded patch set 2 11/27/21, 13:06
-
Sybille Peters updated the commit message in patch set 3 11/27/21, 13:10
-
Sybille Peters uploaded patch set 4 11/27/21, 13:15
-
Sybille Peters marked it ready for review 11/27/21, 13:26
-
Sybille Peters uploaded patch set 5 11/27/21, 13:47
-
core-ci voted on patch set 2 V -1 11/27/21, 14:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19855
-
core-ci voted on patch set 3 V -1 11/27/21, 14:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19856
-
core-ci voted on patch set 4 V -1 11/27/21, 14:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19858
-
core-ci voted on patch set 5 V -1 11/27/21, 14:27 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19861
-
Sybille Peters uploaded patch set 6 11/27/21, 15:36
-
core-ci voted on patch set 6 V +1 11/27/21, 15:56 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19871
-
Stefan Bürk commented on patch set 6 11/27/21, 16:13
Thanks for the patch, nice work, but minor things. Please see my comments, hopefully they explains themself.DatabaseRowsUpdateWizard.php:219'getRowsForTable()' will always return a statement, better check here for rowCount() as early break condition. Another solution would be to check directly in 'getRowsForTable()' the row count and return null of no records are found, which would make this break check also superfluous. That will also make '$hasResults' superfluous in L220, L222 and the break condition at the end. (L281-284)DatabaseRowsUpdateWizard.php:220UnresolvedSuperfluous - see comment above;DatabaseRowsUpdateWizard.php:222UnresolvedSuperfluous - see comment above;DatabaseRowsUpdateWizard.php:284UnresolvedSuperfluous - see comment above;DatabaseRowsUpdateWizard.php:325UnresolvedEventually check here if resultset has records, and if not return 'null' instead, thus early breaking loop in L216, making all further checks as commented above superfluous. Further check if it is a result set (ResultStatement), as we do not want statement here (it should be executed). As that, we cann add real return typehint of '?ResultStatement' here. -
Stefan Bürk commented on patch set 6 CR -1 11/27/21, 18:13
Leave my future comments for now. -1 for now, explanation will follow. -
Stefan Bürk commented on patch set 6 11/27/21, 19:06
Reply Unresolved
-1 stays. I have done 4 test runs with the hog test from patchset 1, which verifies what I have thought and is now proven. ``` MAX_ROWS: 300000 (3 Hunderttausend ) [memory_limit: 2G] ======================================================= Peak memory usage (Start of test): 129MB [LIMIT: 2G] Peak memory usage (After database insertion of 300000 records): 129MB [LIMIT: 2G] Peak memory usage (After row updater): 129MB [LIMIT: 2G] MAX_ROWS: 3000000 (3 Millionen) [memory_limit: 2G] ================================================== Peak memory usage (Start of test): 129MB [LIMIT: 2G] Peak memory usage (After database insertion of 3000000 records): 129MB [LIMIT: 2G] Peak memory usage (After row updater): 129MB [LIMIT: 2G] MAX_ROWS: 10000000 (10 Millionen) [memory_limit: 2G] ==================================================== Peak memory usage (Start of test): 129MB [LIMIT: 2G] Peak memory usage (After database insertion of 10000000 records): 129MB [LIMIT: 2G] Peak memory usage (After row updater): 129MB [LIMIT: 2G] MAX_ROWS: 10000000 (10 Millionen) [memory_limit: 256M] ====================================================== Peak memory usage (Start of test): 129MB [LIMIT: 256M] Peak memory usage (After database insertion of 10000000 records): 129MB [LIMIT: 256M] Peak memory usage (After row updater): 129MB [LIMIT: 256M] ``` Mainloop should and is not memory hungry in the way it is implemented (which was the main topic of this patch / commit message). A very rare condition could be that the database takes lonker than 'max_execution_time' to generate a resultset, even if results are loaded one by one without taking memory. But if someone has so many records, they should either increase the execution time (NOT RECOMMENDED) or do the updates from command line anyway. As doing it from the browser can be a "endless" task. So, it is interessting where your memory issue had come from, maybe from implementing class handler doing row based updates, which holds/gather states ? This should be found and cleaned there, not in the mainloop. Do you have more information or can you retest and debug which is executing and taking the memory ? But changing the mainloop here is not the goal, and I vote against it to change it as it do no really fix the issue, but work around it. Which is not a solution. -
Sybille Peters commented on patch set 6 11/27/21, 19:20
DatabaseRowsUpdateWizard.php:219Reply UnresolvedYes, I know this is awkward. I had already struggled with that. I wanted to get a working solution, but it is not pretty. I was not sure how to safely check if the result has 0 rows. Will look into it further ... I had this before in the getRowsForTable function: if (!$statement || ($statement instanceof Result && $statement->rowCount() === 0) || ($statement instanceof ResultStatement && $statement->columnCount() === 0) || ($statement instanceof Statement && $statement->rowCount() === 0) ) { // empty result return 0; } -
Sybille Peters marked it work in progress 11/27/21, 23:01
-
Benni Mack wrote 11/29/21, 10:19
Change destination moved from master to main -
Benni Mack updated the commit message in patch set 7 11/29/21, 14:06
-
Sybille Peters commented on patch set 7 CR -1 11/29/21, 15:06
Was so far only reproducible on my development system. Seems to be a local issue. Would like to keep as WIP to investigate. -
core-ci voted on patch set 7 V +1 11/29/21, 16:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20089
-
Sybille Peters abandoned the change 12/03/21, 09:44
Memory problem was most likely local issue. Would be interesting to analyze further but should not clutter up core queue