Review main Abandoned

80764[BUGFIX] Ensure minimal sorting order for active packages

Oliver Hader CR 0 V 0 Hard +259 −5 updated 09/19/23, 12:00

Commit message

The order of packages (extensions) in the PackageManager
basically depends on how those extension keys were loaded.
In case no caches are available, the keys are retrieved
from the filesystem in the `scanPackagePathsForExtensions()`.

Besides that, it was observed that there were different
results in the actual TYPO3 installation, in functional test
and in acceptance tests - which revealed some "bias" during
the specific test initializations. OS specifics added on
top of that behavior.

This change tries to introduce a minimal set of ordered
dependencies for active packages. The extension keys `core`,
`backend`, `frontend`, `extbase`, `fluid`, `install` will
always be at the beginning of the list of active packages
(in case they are avaiable in that particular scenario).

The remaining packages are sorted by their corresponding
dependency settings in `ext_empconf.php` properties `depends`
and `suggests` or alphabetically as fall-back.

Context: In a recent scenario, it happened that CSP settings
of `ext:extensionmanager` were processed before `ext:backend`
and `ext:core` were processed.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 29

Review log 2127

Show 12 earlier messages
  1. Oliver Hader uploaded patch set 1 08/30/23, 17:49

  2. core-ci voted on patch set 1 V -1 08/30/23, 18:02 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46058

  1. Oliver Klee commented on patch set 1 08/30/23, 18:33

    PackageManagerTest.php:52

    Please let’s avoid “correctly” in test names, as then the reader cannot know what the correct behavior should be from reading the test name. Instead, I propose this name that states the expected behavior: `activePackagesAreOrderedAlphabeticallyAndByDependency` For guidelines on helpful test names, I recommend this slide: https://speakerdeck.com/oliverklee/test-driven-development-with-phpunit-915a5f2c-3d37-4e41-b5bc-ef1efca9c01e?slide=39
  2. Oliver Hader commented on patch set 1 08/30/23, 18:39

    PackageManagerTest.php:52 Reply

    Thanks for your feedback. This is just a work-in-progress patch... I don't even know whether this is going somewhere. There are too many things that are missing in terms of package dependencies. > `activePackagesAreOrderedAlphabeticallyAndByDependency` That one aspect that is not clear yet in this scope, for instance, just sorting packages alphabetically would not be helpful in the scope of this topic.
  3. Oliver Hader abandoned the change 08/30/23, 18:41

    Giving up here. Too much work would be involved. Dependency ordering produces a flat sequence, but actually should be a nested tree that can be serialized for different strategies (e.g. items without dependencies first or last - depends on the actual use case).
  4. Oliver Hader restored the change 08/31/23, 11:17

  5. Oliver Hader uploaded patch set 2 V -1 08/31/23, 11:21

  6. core-ci voted on patch set 2 V -1 08/31/23, 11:33 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46087

  7. Oliver Hader uploaded patch set 3 V -1 08/31/23, 11:48

  8. core-ci voted on patch set 3 V -1 08/31/23, 11:57 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46090

  9. Oliver Hader commented on patch set 3 08/31/23, 12:53

    The changed order in functional tests is okay... The XML assertions needs to be done on the first level elements (without sorting order) instead of the whole serialized string (with order).
  10. Oliver Hader uploaded patch set 4 V -1 08/31/23, 13:39

  11. Oliver Hader rebased onto patch set 5 08/31/23, 13:45

  12. Oliver Hader commented on patch set 5 08/31/23, 13:49

    This change is ready for review.
  13. core-ci voted on patch set 4 V -1 08/31/23, 13:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46100

  14. core-ci voted on patch set 5 V -1 08/31/23, 14:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46101

  15. Oliver Hader uploaded patch set 6 V -1 08/31/23, 14:15

  16. Oliver Hader commented on patch set 6 08/31/23, 14:21

    Reply

    Done
  17. core-ci voted on patch set 6 V +1 08/31/23, 14:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/46106

  18. Helmut Hummel commented on patch set 6 CR -1 08/31/23, 17:45

    From how I understand the code, the changes and the problem description, I'm not quite sure whether this addresses the issue without negative side effects. Please let me help to better understand this issue and the changes. Thanks

    Unresolved

    Can you please elaborate a bit more on the scenario when this happens? I think some steps to reproduce would be helpful in the forge bug description

    Unresolved

    see above. how exactly can this reproduced? is it in (unit/ functional) tests only? then some change in the testing framework might be more appropriate. Or install tool? Then some changes in FailsafePackageManager might be required. Or is it reproducible with existing PackageStates.php (or in Composer mode), then a more detailed example would be helpful.

    PackageManager.php:634 Unresolved

    Can you explain why you think, this is necessary? Dependency sorting should only be needed, when package states config is not populated yet. Or to put it differently. The packageStatesConfiguration array must be ordered already. getActivePackages is meant as runtime method, which should not call "expensive" sorting functions.

    PackageManager.php:666 Unresolved

    when is this method called, without the packages being populated, that makes this fallback required?
  19. Helmut Hummel commented on patch set 6 08/31/23, 17:52

    I did stop looking at the ext_emconf.php changes after the first three revealed inconsistencies with composer.json. These should for sure be brought in sync again (and maybe revised generally)

    ext_emconf.php:16 Unresolved

    I don't know when these files diverged from the composer.json, but to be technically correct, the dependencies here must be identical to the dependencies declared in the composer.json file

    ext_emconf.php:15 Unresolved

    if a dep is added here, it must also be added to composer.json, otherwise the ordering in composer and non composer mode might be different

    ext_emconf.php:15 Unresolved

    same as belog

    ext_emconf.php:15 Unresolved

    same as adminpanel
  20. Oliver Klee commented on patch set 6 08/31/23, 18:06

    Unresolved

    Maybe we can add the missing dependencies first in a pre-patch to keep the changes smaller, easier to review and more focused.
  21. Oliver Hader commented on patch set 6 CR -2 08/31/23, 18:16

    Thanks Helmut, I get back to you via Slack...
  22. Oliver Hader marked it work in progress 08/31/23, 18:17

  23. Markus Klein commented on patch set 6 08/31/23, 20:55

    Unresolved

    Without knowing the details, but reading the actual problem to be solved in the commit message: Please do not infer any order for any feature from extension loading order except PHP class dependencies. The extension ordering is way too general to derive any order for any other feature/configuration/etc. without running into issues again, maybe later. Please consider introducing a dedicated sorting mechanism (possibly suing the DependencyOrderingService) for CSP instead of trying to "fix" the extension loading order. Just my 2c
  24. Oliver Hader commented on patch set 6 09/19/23, 12:00

    → moved to https://review.typo3.org/c/Packages/TYPO3.CMS/+/81076 which just tackles the testing-framework
  25. Oliver Hader abandoned the change 09/19/23, 12:00