Review main Abandoned

84376[TASK] Save RAM in PHPUnit functional tests

Ayke Halder CR 0 V 0 Normal +118 −0 updated 06/17/24, 13:45

Commit message

This configures PHPUnit to control the garbage collector. Without this
the garbage collector was executed by PHP itself on running into
`memory_limit` - and not properly used for an unlimited `memory_limit`.

This implements a missing `tearDown` method to selected PHPUnit
functional tests which persist to much RAM.

In `tearDown` the properties set in `setUp` are unset again and thereby
do not pollute the RAM of the running test-space.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 23

Show 3 more files

Review log 2331

Show 46 earlier messages
  1. Ayke Halder uploaded patch set 1 05/23/24, 17:21

  2. core-ci voted on patch set 1 V +1 05/23/24, 17:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58110

  3. Stefan Bürk commented on patch set 1 05/23/24, 21:37

    Thank you for the patch. However, when stating things like "performence increasment" or "save mem usage" it's kind of good to state some numbers to give reviewes a kind of feeling or the impact. I have done locally the measurement without(before) and with this change(after): ``` UNIT ---- BEORE.......: Memory: 465.06 MB AFTER.......: Memory: 467.06 MB FUNCTIONAL ---------- BEORE.......: Memory: 2.14 GB AFTER.......: Memory: 1.65 GB ``` which reaveals relativly no impact on unit test level, but for the functional tests it is in a remarkable area, so it could and should be considered. However,instead of simply unsetting the subject or related testcase based properties in tear-down I would prefere to remove the proepries all together and create the subject/related services in the test cases directly as test method internal variable - eventually having kind of a private `createServiceXy()` method to be called. Removing the properties completly and avoiding the need to add tearDowns() just for the unset calls. What do you think ?
  4. Garvin Hicking updated the commit message in patch set 2 V +1 05/24/24, 12:36

  5. core-ci voted on patch set 2 V +1 05/24/24, 12:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58159

  6. Christian Kuhn uploaded patch set 3 V +1 05/26/24, 13:53

  1. core-ci voted on patch set 3 V +1 05/26/24, 14:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58284

  2. Christian Kuhn commented on patch set 3 05/26/24, 14:00

    Ok. I removed the unit test tearDown() changes from the patch for now: When I last measured this, I ended up removing a general "unset all local properties in tearDown()" in TF with https://github.com/TYPO3/testing-framework/commit/f55c7154e48c48d956a656fd1e8ebd60d7a515cf - it actually increased memory usage. I'm unsure *why*, but the same happens with the new unset()'s of patch set 2 in unit tests, now. I'd guess that has todo with garbage collection internals that I don't understand.
  3. Christian Kuhn commented on patch set 3 05/26/24, 14:30

    imho, the general issue with this is that phpunit keeps references to test cases after tests have been executed. i *think* this is because phpunit pulls the results from test cases at the very end of the run, so it needs to keep test case instances. this is why memory consumption essentially increases with each test case: the garbage collection can not remove test case instances when they are still referenced. i think this is a long standing issue in phpunit. i'll try to chat on this with sebastian bergman - he'll join the next core sprint, which gives us a good chance to do so.
  4. Ayke Halder uploaded patch set 4 V +1 05/26/24, 16:30

  5. core-ci voted on patch set 4 V +1 05/26/24, 16:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58286

  6. Ayke Halder uploaded patch set 5 V +1 05/26/24, 16:49

  7. core-ci voted on patch set 5 V +1 05/26/24, 16:53 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58287

  8. Ayke Halder commented on patch set 4 05/26/24, 17:13

    Reply

    Let's see what Sebastian Bergman thinks about this. The docs say to use `setUp`/`tearDown`: https://docs.phpunit.de/en/10.5/fixtures.html#fixtures > PHPUnit supports the reuse of setup code between tests. Before a test method is run, a template method called `setUp()` is invoked: this is where you can create your test fixture. Once the test method has finished running, whether it succeeded or failed, another template method called `tearDown()` is invoked: this is where you can clean up the objects against which you tested. If we do not use `setUp`/`tearDown` and use private methods like `createServiceXy()` instead, we have to add extra lines to each `#[Test]` method. E.g. for `ContentObjectRendererTest` this would add about 3 lines to each of the 156 tests: ``` $subject = createSubject(); $frontendControllerMock = createFrontendControllerMock(); $cacheManagerMock = createCacheManagerMock(); ``` -> ~450 extra lines - I don't like this.

    Reply

    Please see https://github.com/TYPO3/testing-framework/pull/562 In `tearDown` method all properties of the test-object are unset to free up memory. This memory is only reclaimed if neccessary, e.g. the memory of the running PHP process is running into its limitation and thereby an automatic garbage collection is triggered. The limit for memory is really high, therefor garbage collection is never triggered and the memory adds up. To use as little memory as possible we must manually force a garbage collection. As garbage collection consumes time this should be done only once per test-set in `tearDownAfterClass` method, instead of running it for every single test in `tearDown`. So we must introduce garbage collection in `tearDownAfterClass` to make this PR work!
  9. Stefan Bürk commented on patch set 5 05/26/24, 21:32

    Reply

    I don't like to use such properties. We already remove the one or other as side changes when redoing stuff on tests. Creating a subject in setUp (with the tearDown cleanup) is a suggestion, a possible way and not a single-way-of-truth mantra. If we can avoid to introduce property checking and garbace collector calls by moving to method instead of introducing additional handling code in test execution using code reflections on the classes - I would prefere the way to avoid these properties in the first place. On the other hand I would ask why we should implement something in the testing-framework to mitigate this, if this is PHPUnit way to go. I postulate, that garbage collection should be part of PHPUnit then - or not ?
  10. Ayke Halder commented on patch set 5 05/27/24, 07:09

    Reply

    If we like to go this way and 'avoid properties': To make it future prove we would need to somehow enforce it for test cases. Otherwise people looking into the base class `FunctionalTestCase` or `UnitTestCase` will just use properties as these are also implemented there. > If we can avoid to introduce property checking and garbace collector calls by moving to method … Also with this method approach we might still need to call the garbage collector. > garbage collection should be part of PHPUnit then - or not ? There are multiple ways to use the testing framework. E.g. using one-process-per-test does not need to call the GC - as the the whole process-memory is freed on process termination.
  11. Stefan Bürk commented on patch set 5 05/27/24, 07:16

    Reply

    I related to additional subject/mock object properties, not the control properties. It's already considered to move the direct property setting into constroctor call anyway - and control properties needs to be kept. https://review.typo3.org/c/Packages/TYPO3.CMS/+/84414 could me such integrity check for core internal custom proeprty usages.
  12. Ayke Halder commented on patch set 5 05/27/24, 07:47

    Reply

    > Otherwise people looking into the base class FunctionalTestCase or UnitTestCase will just use properties as these are also implemented there. I mean 'will just implement their own properties'. > https://review.typo3.org/c/Packages/TYPO3.CMS/+/84414 Looks promising. But it will take some time to refactor all the test-cases. How about using a general property-unset in the testing-framework for now? As this only works for public/protected properties but not for private properties an additional 'test class private property integrity checker' in checkIntegrityPhp will do. Refactoring all properties in all test-cases from private to protected once is do-able.
  13. Ayke Halder uploaded patch set 6 V +1 05/27/24, 08:36

  14. core-ci voted on patch set 6 V +1 05/27/24, 08:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58327

  15. Ayke Halder commented on patch set 5 05/27/24, 08:41

    Reply

    I've got the garbage collector runninng by configuring ``` <phpunit controlGarbageCollector="true"/> ``` To sum up the result: * This enableds garbage collection, * This adds unset for properties in tearDown for the most needed tests and thereby saves about 0.5 GB of RAM in functional tests. All other ideas about not using properties can be implemented in the future.

    Reply

    Execution of garbage collector fixed by setting ``` <phpunit controlGarbageCollector="true"/> ```
    This is working for now as expected.
  16. Ayke Halder updated the commit message in patch set 7 V +1 05/27/24, 08:44

  17. core-ci voted on patch set 7 V +1 05/27/24, 08:49 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58328

  18. Oliver Klee updated the commit message in patch set 8 V +1 05/27/24, 09:04

  19. core-ci voted on patch set 8 V +1 05/27/24, 09:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58333

  20. Christian Kuhn commented on patch set 8 CR -1 05/27/24, 09:16

    i'm in general very reluctant on this patch: i for instance played with ExportTest and figured it actually had no measurable impact adding unset(). we should at least drill down the patch to those classes that really trigger measurable impact, and then see if they could just be simplified to avoid the properties in question entirely. i also don't like the idea to force protected over private methods - it's a regression codewise. i think this patch should be marked [WIP].

    FunctionalTests.xml:23

    i'm not convinced TF should deal with gc. this is a general phpunit issue that should be solved on this level if really needed.

    DataHandlerTest.php:66

    i removed those since it had no measurable impact. you added them again. how to proceed?

    ExportTest.php:81

    see https://review.typo3.org/c/Packages/TYPO3.CMS/+/84421 as an example on how to systematically improve such a test and have for instance less mocking along the way.
  21. Christian Kuhn commented on patch set 8 05/27/24, 09:33

    note: i don't want to kill this patch entirely. i just think we may not have all the information at hand we need to come up with a good decision. i think we should take the opportunity next sprint to check with sebastian bergmann on this topic as well, to see which good practices he may have in mind. in the meantime, i think it would be good if we see which test cases in particular are memory greedy - we should imho pick something like the majority of this half gig of saved ram, and have a dedicated patch for this - my expectation is, that will be 1 or two test cases only to look at. maybe a removal of the properties is possible for these big ones?
  22. Ayke Halder commented on patch set 8 05/27/24, 09:45

    Reply

    Please tell me how you measure. I measure using this: https://github.com/TYPO3/testing-framework/pull/562 (Yes, this measurement influences the result as it calls the garbage collector. And you must remove the `unset()` in `tearDown()` from the testing-framework patch to make these changes here become effective in comparison.) Without change: ``` # TYPO3\CMS\Impexp\Tests\Functional\ExportTest !!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!.!!. Memory leak: 30.69 MB ``` With change: ``` # TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory leak: 2.29 MB ``` > we should at least drill down the patch to those classes that really trigger measurable impact, Already did this. And here we are with all the classes that have a bigger impact than 30 MB on memory. Please have a look: https://forge.typo3.org/issues/103889#note-3

    DataHandlerTest.php:66 Reply Unresolved

    Please tell me how you measure. * Running a single functional test only does not give correct memory value. * Running all functional tests and then see the memory does. The output "Memory" from PHPUnit is missleading (especially for a single test). It shows the max memory used or something similar - not additional memory. additional memory is the memory that stays occupied by this one test after the test has finished when it is run with all the other tests together. I'm looking into "additional memory" for each test.
  23. Ayke Halder commented on patch set 8 05/27/24, 15:34

    Reply Unresolved

    Potential memory savings by test: * 102.10 MB `\TYPO3\CMS\Frontend\Tests\Functional\ContentObject\FilesContentObjectTest` * 101.25 MB `\TYPO3\CMS\Core\Tests\Functional\DataHandling\DataHandler\SecurityTest` * 73.90 MB `\TYPO3\CMS\Core\Tests\Functional\Imaging\IconFactoryTest` * 54.30 MB `\TYPO3\CMS\Extbase\Tests\Functional\Persistence\RelationTest` * 51.76 MB `\TYPO3\CMS\Backend\Tests\Functional\Authentication\PasswordResetTest` * 49.31 MB `\TYPO3\CMS\Backend\Tests\Functional\Controller\MfaConfigurationControllerTest` * 30.69 MB `\TYPO3\CMS\Impexp\Tests\Functional\ExportTest` * 20-29 MB 5x Tests * 10-19 MB 8x Tests * 5-9 MB 10x Tests Where shall we do the cut?

    FunctionalTests.xml:23 Reply

    ExportTest.php:81 Reply Unresolved

    This example still is in need for a properly controlled garbage collection. Run the test-case 5 times in a row: ``` ./Build/Scripts/runTests.sh -s functional -p 8.2 -d mariadb typo3/sysext/impexp/Tests/Functional/ExportTest.php typo3/sysext/impexp/Tests/Functional/ExportTest.php typo3/sysext/impexp/Tests/Functional/ExportTest.php typo3/sysext/impexp/Tests/Functional/ExportTest.php typo3/sysext/impexp/Tests/Functional/ExportTest.php ``` #### Without GC: Results in 78.27 MB of unfreed memory after last test. ``` Using driver: mysqli PHPUnit 11.0.3 by Sebastian Bergmann and contributors. Runtime: PHP 8.2.19 Configuration: /home/ayke/PhpstormProjects/Typo3/TYPO3.CMS/Build/phpunit/FunctionalTests.xml # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: +64.82 MB / 73.30 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: -3.77 MB / 69.53 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: +6.22 MB / 75.75 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: -3.29 MB / 72.47 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!. 65 / 70 ( 92%) !.!.!.!.!. 70 / 70 (100%) Memory: +5.80 MB / 78.27 MB Time: 00:16.932, Memory: 103.00 MB OK (70 tests, 125 assertions) ``` #### With GC in `tearDownAfterClass`: Results in 70.51 MB of unfreed memory after last test. ``` # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: +60.70 MB / 69.18 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: +0.35 MB / 69.53 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: +0.29 MB / 69.82 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!.!.!.!.!.!. Memory: +0.35 MB / 70.16 MB # \TYPO3\CMS\Impexp\Tests\Functional\ExportTest !.!.!.!.!.!.!.!.!. 65 / 70 ( 92%) !.!.!.!.!. 70 / 70 (100%) Memory: +0.35 MB / 70.51 MB Time: 00:17.068, Memory: 103.00 MB OK (70 tests, 125 assertions) ```
  24. Ayke Halder commented on patch set 8 05/27/24, 16:46

    As already mentioned in commit message: This configures PHPUnit to control the garbage collector.

    FunctionalTests.xml:23 Reply

    https://github.com/TYPO3/testing-framework/pull/562#issuecomment-2133707768 @lolli@schwarzbu.ch: > … I came to essentially the same solution phpunit obviously has already: Trigger gc once in a while! > I don't agree with Ayke gc should be run after each test case: gc() needs to trade runtime with memory. memory is not top-1, we don't need to force "lowest memory usage possible" at all costs. We want a good trade-off instead. So doing it every x tests is imho a really good idea, and we should play around with these settings!
  25. Ayke Halder abandoned the change 06/17/24, 13:45

    PHPUnit has implemented a solution to significantly reduce PHPUnit's test memory usage.