Review main Abandoned

73196[WIP][FEATURE] Add configuration object system

crell CR 0 V 0 Nightmare +2367 −484 updated 08/01/22, 18:05

Commit message

Proof of concept for now.

This uses the new Serde library to expose defined
configuration objects to the rest of the system through
DI.  Config objects are loaded from YAML files if
available, with a fallback to read the existing
TYPO3_CONF_VARS array if not found. The config
object is also cached to a PHP file on disk.

It then also updates Mailer and related classes
accordingly, as a demonstration. A few other
config objects are included for demonstration
but not fully converted yet.

It now also includes a prototype of attributes
to define a form based on a class, too.
It's entirely parallel to serialization but
uses the same attribute engine. These could
be used to auto-generate forms based on any
class that is so-attributed.

Used commands:
composer req "symfony/validator" ^5.4
composer req "symfony/validator" ^5.4 \
-d typo3/sysext/core --no-update
composer req "crell/fp" ^0.3.3
composer req "crell/fp" ^0.3.3 -d typo3/sysext/core --no-update
composer req "crell/attributeutils" ^0.6.1
composer req "crell/attributeutils" ^0.6.1 \
-d typo3/sysext/core --no-update
composer req "crell/serde" ^0.3.2
composer req "crell/serde" ^0.3.2 \
-d typo3/sysext/core --no-update
composer req "crell/serde" ^0.3.2 \
-d typo3/sysext/install --no-update

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 42

Show 22 more files

Review log 4265

Show 2640 earlier messages
  1. crell uploaded patch set 1 01/28/22, 22:22

  2. core-ci voted on patch set 1 V -1 01/28/22, 22:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22641

  3. Stefan Bürk commented on patch set 1 01/28/22, 23:40

    some early/first notes btw. add the pre-commit hook which checky cgl locally - at least run now the fix locally to fix the cgl issues ;)
    again: main, not master.

    GfxConfig.php:22

    main is min PHP8.1 - so todo can directly solved ;) - use "constructor property promotion" to eliminate the properties and make them readonly directly

    Http.php:25

    What are these attributes (for) ? Is this where you need one of your libs/packages you have been building ? If not used yet, should be avoided - otherwise include your package and demonstrate it ...
  4. Stefan Bürk updated the commit message in patch set 2 01/29/22, 00:15

  5. core-ci voted on patch set 2 V -1 01/29/22, 00:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22646

  6. crell marked it work in progress 01/29/22, 17:42

  7. crell commented on patch set 2 01/29/22, 17:44

    This change is ready for review.

    Reply

    Forgot to mark WIP on this. I thought it would go to the previous issue. Please ignore for now, it's just a rebase of the old patch so I can start working on the Serde-based version.
  8. crell uploaded patch set 3 02/01/22, 00:14

  9. core-ci voted on patch set 3 V -1 02/01/22, 00:16 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22842

  10. crell updated the commit message in patch set 4 02/01/22, 00:16

  11. core-ci voted on patch set 4 V -1 02/01/22, 00:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22843

  12. crell commented on patch set 4 02/01/22, 19:41

    Reply

    Ack

    GfxConfig.php:22 Reply

    Done

    Http.php:25 Reply

    Done
  13. Oliver Klee commented on patch set 4 02/01/22, 20:16

    In general, please let's note have a period at the end of commit message subject lines: https://chiamakaikeanyi.dev/how-to-write-good-git-commit-messages/
  14. crell uploaded patch set 5 02/01/22, 20:21

  15. core-ci voted on patch set 5 V -1 02/01/22, 20:23 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22893

  16. Stefan Bürk commented on patch set 5 02/01/22, 20:30

    * remove: "Build/phpunit/.phpunit.result.cache" which MUST not be in the repo, also this SHOULD not created anymore, so delete it !!! (and remove it from the patch). * rebase your patch on current main solving merge conclict: see https://docs.typo3.org/m/typo3/guide-contributionworkflow/main/en-us/Setup/Git/Index.html#set-autosetuprebase and https://docs.typo3.org/m/typo3/guide-contributionworkflow/main/en-us/HandlingAPatch/Rebase.html#method-2-git-pull-rebase * Either add "[POC]" to commit topic line and/or switch to WIP mode - or remove "Proof of concept from commit message" * again, use either cgl script manually or in combination as pre-commit hook and fix cgl before pushing - would help you to not have failing pipelines. At least not for the code file parts. See: https://docs.typo3.org/m/typo3/guide-contributionworkflow/main/en-us/Setup/Git/Index.html#pre-commit-hook > phpunit result cache has been disabled a longer time ago, and thus removed from .gitignore. Cleanup your coredev installation It's confusing is this patch is ready for real review or if it still is in work.
  17. Stefan Bürk commented on patch set 5 02/01/22, 20:31

    Reply

    "Resolving" directly was not intended. So reopen it.
  18. Oliver Klee wrote 02/01/22, 20:57

    Removed cc Oliver Klee.
  19. crell marked it work in progress 02/01/22, 23:46

  20. crell commented on patch set 5 02/01/22, 23:48

    This change is ready for review.

    Reply

    I keep setting this to WIP. I don't know why it's getting overridden. There's nothing to see here yet. Please ignore for now. And I have no idea why the result cache ended up in the patch... Shouldn't that be in gitignore?
  21. Stefan Bürk commented on patch set 5 02/02/22, 02:33

    Reply

    That file should not be there at all ! phpunit configuration for core is setup to not creating/using a result file at all. So it should not be written pysically. Pls. delete that file AND remove it from the patch. After removing from patch ensure that file is absolutly killed in your instance/filesystem. Do you run phpunit with a custom configuration and not the core one / Build/Scripts/runTests.sh ?
  22. crell uploaded patch set 6 02/02/22, 20:41

  23. core-ci voted on patch set 6 V -1 02/02/22, 20:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22924

  24. crell commented on patch set 6 02/02/22, 20:44

    Reply

    Ack

    Reply

    Ack
  25. crell uploaded patch set 7 02/03/22, 18:02

  26. core-ci voted on patch set 7 V -1 02/03/22, 18:05 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22966

  27. crell uploaded patch set 8 02/03/22, 22:23

  28. core-ci voted on patch set 8 V -1 02/03/22, 22:35 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22980

  29. crell uploaded patch set 9 02/03/22, 23:38

  30. core-ci voted on patch set 9 V -1 02/03/22, 23:40 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22984

  31. crell uploaded patch set 10 02/08/22, 21:08

  32. core-ci voted on patch set 10 V -1 02/08/22, 21:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23240

  33. crell uploaded patch set 11 02/08/22, 21:11

  34. core-ci voted on patch set 11 V -1 02/08/22, 21:19 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23242

  35. crell uploaded patch set 12 02/09/22, 16:03

  36. core-ci voted on patch set 12 V -1 02/09/22, 16:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23291

  37. crell uploaded patch set 13 02/09/22, 20:46

  38. core-ci voted on patch set 13 V -1 02/09/22, 20:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23304

  39. crell uploaded patch set 14 02/14/22, 15:16

  40. core-ci voted on patch set 14 V -1 02/14/22, 15:29 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23525

  1. crell uploaded patch set 15 02/14/22, 16:56

  2. core-ci voted on patch set 15 V -1 02/14/22, 17:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23535

  3. Benni Mack commented on patch set 15 02/14/22, 17:20

    Just added a few comments while reading through the code first time. One thing I do not understand yet: I guess the config objects are built during runtime (= when needed), because we don't know if they will change through ext_localconf.php etc. which makes sense. However, if we actually use the objects a lot (imagine "FE" is used in some objects A LOT, e.g. creating such an object for each link that is generated in PageLinkBuilder for example, when you search for TYPO3_CONF_VARS[FE]) then we somehow should re-use the objects that are built (but only after the boot-state = after all extensions have loaded ext_localconf.php) in the container. right?

    GfxConfig.php:29

    how would one deprecate properties like this in the future? I think getters do their job in this regard to easily move forward when removing / deprecating properties. One example would be $processorPathLzw which does not serve any "real" use except for working with TIFF files.

    ConfigDefinitionPass.php:59

    looks like a lot of magic here :) pipe()? Unsure if we should introduce fp to TYPO3 code as a "side-effect" with this change.

    ServiceProvider.php:102

    are we going to have "serde" wired for everybody available in TYPO3? If so, I think we should go with a stable version / API here. We do use packages like "phpstan" which did not have a major version yet, but these are just dev dependencies unlike serde.

    FileListController.php:32

    I'd recommend if we introduce object-based configuration reading, we should name it in a more speaking fashion than the cryptic naming. So I suggest using "\TYPO3\CMS\Core\Configuration\GraphicsConfiguration"
  4. crell uploaded patch set 16 02/14/22, 17:34

  5. Benjamin Franzke commented on patch set 16 02/14/22, 17:48

    phpstan.neon:39

    This is for sure a opinionated "error" that phpstan repots here, still I agree to the phpstan authors that this is code smell and I'm against introducing readonly properties that aren't initialized in the constructor. This isn't properly statically analyzable, therefore it's actually better we avoid it.

    ConfigDefinitionPass.php:54

    All this compiler pass does is plain definitions of config classes. This could basically just be the following in Services.yaml: TYPO3\CMS\Core\Config\GfxConfig: factory: ['@TYPO3\\CMS\\Core\\Config\\ConfigFactory', 'getConfigClass'] arguments: ['GFX', 'TYPO3\CMS\Core\Config\ConfigFactory'] Doesn't seem reasonable for me to introduce two new composer dependencies and a compiler pass just to avoid that tiny bit of (actually very readable and introspectable) boilerplate in Services.yaml

    MemorySpool.php:71

    no, use makeInstance, it will always use getContainer (which is internal and shouldn't be used here).

    ServiceProvider.php:112

    Such Utility function services no purpose in this kind of a service provider. The `core` ServiceProvider is a service provider for the core extension, not a general utility-trash-bin.

    Services.php:11

    Please follow the existing coding style. use-statements are avoided if relative class-references are possible in this file.

    Services.yaml:357

    redundant with definition in core/Classes/ServiceProvider.php

    Services.yaml:369

    All the "class: …" definitions added here are redundant and therefore superfluous.

    composer.json:22

    dependency unused in EXT:install, therefore superfluous in this composer.json
  6. core-ci voted on patch set 16 V -1 02/14/22, 17:48 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23541

  7. crell commented on patch set 16 02/14/22, 17:52

    Reply

    The config objects are exposed as services, so the container will cache them like any other object already. The first time it's requested the factory will build it and return it, and any subsequent requests will just pull it out of the container's existing cache.

    GfxConfig.php:29 Reply Unresolved

    I haven't fully modeled Gfx yet; this is just a dumb-straight-conversion at the moment. In general, though, there's a couple of ways to migrate properties. 1. Serde supports aliases, so you can have a property foo that will pull from the "foo" key in the incoming data (array in this case), and if not found will pull from "bar". 2. The constructor can set $this->bar = $this->foo to create an alias, if we rename bar to foo at some point. Both can be readonly, but only one promoted (the new one, presumably). 3. If #2 is too clumsy or clunky in a specific case, we can also lazily instantiate a readonly property like so: https://peakd.com/hive-168588/@crell/php-tricks-lazy-public-readonly-properties I think that should be enough options to avoid otherwise-redundant getter methods in most cases.

    ConfigDefinitionPass.php:59 Reply Unresolved

    It's utilities in the Crell/fp library that Serde already uses, so it's pulled in either way. I've used it a few times in this code because it's about 1/4 the size of trying to do map, filter, and index-by the slow way, and honestly it's also easier to read once you get used to it. We could do it the foreach()-way, but at least for the moment I went with the way that seemed more ergonomic and compact.

    ServiceProvider.php:102 Reply Unresolved

    I don't have a 1.0 of Serde yet, mainly because I'm not 100% certain that some additions won't require a small API change, and in the process of writing these patches I'm spotting an edge case or two in some of the dependencies. (The 0.3.1 release was to account for a bug fix in Crell/fp that this patch found.). It should be fairly stable though, certainly the main front-end usage. (The only thing I can think of to maybe change there would be optional serialization groups, which would only be an optional extra argument.) I can call it 1.0 at whatever point we are comfortable calling it that, though.

    FileListController.php:32 Reply

    Fine by me. I will rename it to GraphicsConfig in the next patch. I have been using Config as a suffix for now since "Configuration" is already used. I don't have particularly strong feelings about that, though, so I'm OK with most consensus options. (A good IDE will make renaming it to whatever we go with trivial.)
  8. crell commented on patch set 16 02/14/22, 18:11

    phpstan.neon:39 Reply

    Forbidding RO properties not assigned in the constructor would forbid a large number of very valid, and for serialization necessary actions. Specifically, exposing reflection to the attributes simply cannot be done in the constructor. The alternative is to not use readonly properties at all, which would mandate redundant getter methods for every single property, something PHP 8.1 is specifically designed to make unnecessary. I don't see any value in SA dogmatism here if it means making every config class 5x larger. Not every technical code smell is actually smelly. I firmly believe phpstan is incorrect on this one, and the ergonomic cost of following that overly-strict opinion is far too high. It cuts off half the use cases for readonly properties.

    ConfigDefinitionPass.php:54 Reply Unresolved

    If it were 3-4 services I'd probably agree. However, the intent is that it would be a much larger number, and would grow as we split up the deep TYPO3_CONF_VARS arrays and extensions are added and include their own config. So we'd be looking at dozens, in practice. For that, having the classes auto-detect and register automatically is a much better experience for extension developers. It doesn't look like it now, because getClassList() is hard coded. The intent is to make that dynamic by scanning the file system, but I'm not sure how to do that yet. It also makes it a bit easier to change the registration mechanism, which right now is still in development. (I have to add the YAML file lookup yet.)

    MemorySpool.php:71 Reply Unresolved

    I ran into problems with that before, which is why I changed it. I can't seem to replicate it now, so I'll try switching it back and see if the bot takes issue with it.

    ServiceProvider.php:112 Reply Unresolved

    This is a little vestigial at the moment. The HTTP config is used during the installer, so it does need to be in ServiceProvider. Likely others are too. I had the Http config object available here previously, but then realized that right now Http has a property that is expected to be callables, which... I don't know what to do with at the moment, so I backed that out. I just didn't back out these pieces. However, this code or something akin to it *will* be necessary for config objects to be available in the installer.

    Services.php:11 Reply

    Ack

    Services.yaml:357 Reply Unresolved

    Per other comment, which one should I remove for now?

    Services.yaml:369 Reply Unresolved

    How do you indicate to the container to load that class, then? It's not in an auto-wired directory, and having the class-name-as-service without any keys under it is invalid YAML syntax, no?

    composer.json:22 Reply Unresolved

    As above, technically true right now but won't be eventually. (Also, is the guideline for what to depend on where documented somewhere? I've mostly just seen "run these lines" recommendations, which include install.) I could back it out for now, but it will come back sooner or later.
  9. crell uploaded patch set 17 02/15/22, 16:48

  10. core-ci voted on patch set 17 V +1 02/15/22, 16:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23642

  11. crell uploaded patch set 18 02/16/22, 00:03

  12. core-ci voted on patch set 18 V +1 02/16/22, 00:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23697

  13. crell uploaded patch set 19 02/16/22, 22:50

  14. core-ci voted on patch set 19 V -1 02/16/22, 23:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23763

  15. crell commented on patch set 19 02/16/22, 23:03

    I think this is now in a finally reviewable state. :-) Some notes: * As noted in some of the previous comments, I included some changes to ServiceProvider that will be necessary to expose config objects to the installer. I added them as part of the HTTP config object, but had to back that out when I realized it had union types, which I don't know how to deal with yet. It's not used currently, but will be eventually. * I am not using the core PHP cache at the moment, as it's currently configured BY the config system, creating a circular dependency. There may be a way to break that, and I'm fine with switching if we can, but this works for the moment. It's not necessarily final. * There's a few @todos in the code for places that are still known rough or are pending some other work first. * I originally tried to use a type map to fold all of the transport-specific objects into a single property with a common interface. I forget why exactly I ran into a wall there but it didn't work properly, so I backed that out. There are other things we can do to fiddle with property names to make them more readable quite easily, thanks to Serde, but that's something we can bikeshed later. * The location/namespace/name of various classes is probably wrong. "Configuration" is already taken. I'm flexible on what pattern we want to go with here. * Still pending is figuring out how to detect config classes. I do not believe it's a good DX for extension developers to manually register them in the container themselves. It's much better to detect them by location or attribute, but I'm not sure of the best way to do that at the moment. Suggestions welcome. * There is no UI or write-back support yet, as expected.

    FileListController.php:32 Reply

    Done
  16. crell uploaded patch set 20 02/16/22, 23:09

  17. core-ci voted on patch set 20 V -1 02/16/22, 23:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23764

  18. crell uploaded patch set 21 02/16/22, 23:46

  19. core-ci voted on patch set 21 V +1 02/16/22, 23:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23767

  20. Benjamin Kott commented on patch set 21 02/17/22, 10:19

    Questions: 1. How can I dump the default config for reference? 2. How can I dump the current config? 3. Why is the config within for example mail.yaml not prefixed? General: 1. All directly used packages must be defined as root requirement 2. ... developing

    phpstan.neon:43 Unresolved

    I would prefer having each and every occurrence explicitly ignored if nessesary. Even it´s an php stan error, phpstan will then also complain if the error does not occur as soon as the problem is resolved.

    composer.lock:122 Unresolved

    direclty used dependencies are not declared in root composer.json

    composer.lock:186 Unresolved

    direclty used dependencies are not declared in root composer.json

    ConfigFactory.php:53 Unresolved

    needs to have a follow up/ticket afterwards so it does not get lost.

    ConfigFactory.php:64 Unresolved

    unresolved todo, iirc - you cannot this needs to be protected by default through the packaged htaccess, IIS config and adjusted documentation for nginx usage. NOTE: this is only the case for "classic" non composer installations, for composer installations the var/cache directory is outside the root.

    ConfigFactory.php:81 Unresolved

    how can the cache be cleared? i guess we are not using the caching framework because it will at some point also handle the config for the caching framework?

    GraphicsConfig.php:23 Unresolved

    why do we not implement an interface that forces the hydratable?

    MailConfig.php:51 Unresolved

    how is validated that the array keys are integers to be sortable? does that mean we have to reset all values if we only want to extend it? so how does our config look like? a) mail: layoutRootPaths: 30: EXT:superduper/Resources/Private/Layouts/ b) mail: layoutRootPaths: 0: EXT:core/Resources/Private/Layouts/ 10: EXT:backend/Resources/Private/Layouts/ 20: EXT:superduper/Resources/Private/Layouts/

    MailConfig.php:58 Unresolved

    this is ultra hard to read because of missing empty lines after each argument

    MailConfig.php:76 Unresolved

    can we export the comment block? otherwise the documentation is full of html markup only makes it hard to read

    MailConfig.php:200 Unresolved

    the config object should not contain utility validation methods. the config object should not be able to be constructed with invalid settings also consider `validateEmail` or `isValidEmail` for future implementations

    MailConfigDsn.php:7 Unresolved

    MailDsnConfig?

    MailConfigMbox.php:10 Unresolved

    MailMboxConfig?

    MailConfigSendmail.php:10 Unresolved

    MailSendmailConfig?

    MailConfigSmtp.php:11 Unresolved

    MailSmtpConfig?

    MailerTest.php:82 Unresolved

    code is commented out, can be removed

    MailerTest.php:94 Unresolved

    code is commented out, can be removed

    ConfigFactoryTest.php:47 Unresolved

    update serde please and replace, as benni mentioned if serde is not stable i would not consider it usable for now.

    ConfigFactoryTest.php:168 Unresolved

    i am not a fan of that yaml config definitions are not prefixed and the filename seems to be used as context identifier

    MailConfigSmtpTest.php:10 Unresolved

    MailSmtpConfigTest?
  21. crell uploaded patch set 22 02/18/22, 00:52

  22. core-ci voted on patch set 22 V -1 02/18/22, 01:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23846

  23. crell uploaded patch set 23 02/19/22, 00:24

  24. core-ci voted on patch set 23 V -1 02/19/22, 00:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23893

  25. crell abandoned the change 08/01/22, 18:05