78953[TASK] Add migration section to Changelog
Commit message
Review
-
Code review CR 0
-
Verified V 0
Issues 1
Files 1
Review log 2530
Show 45 earlier messages
-
Sybille Peters uploaded patch set 1 05/06/23, 07:59
-
core-ci voted on patch set 1 V +1 05/06/23, 08:06 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41626
-
Oliver Klee commented on patch set 1 05/06/23, 09:59
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:113I propose both lists to unordered lists as the order of the items does not matter in this case (as far as I can tell). -
Sybille Peters commented on patch set 1 05/06/23, 10:19
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:113ReplyAgreed. Will fix. -
Sybille Peters uploaded patch set 2
V +105/06/23, 10:22
-
Sybille Peters commented on patch set 2 05/06/23, 10:23
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:104ReplyDoneFeature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:113ReplyDone -
core-ci voted on patch set 2 V +1 05/06/23, 10:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41628
-
Oliver Klee voted on patch set 2 V +1 CR +1 05/06/23, 10:52
-
Helmut Hummel commented on patch set 2 CR -1 05/06/23, 11:08
Thanks. Makes sense to add more things here. I have some comments thoughFeature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:111For Composer projects, not sure if this is relevant. I would rephrase, that all extensions MUST (TYPO3 12, in 11 it is "SHOULD") be installed with Composer to be recognized as TYPO3 ExtensionsFeature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:115This is not correct. It can be of any type. If it is typo3-cms-extension TYPO3 will properly recognize and activate it.Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:127isn't this unrelated to this change and not yet enforced in non Composer mode?Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130I don't understand this pointFeature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:132that has not changed at all, thus should not be mentioned here -
Sybille Peters uploaded patch set 3
CR +1V +105/06/23, 15:04 -
Sybille Peters commented on patch set 2 05/06/23, 15:04
General info: I hope to make it easier for extension developers. What I wrote in "Migration" mostly summarizes what is already written above. I realize though that there were several patches with related changelogs so the information might already be there somewhere. We are also updating the documentation (composer.json and Extension Testing pages), there is some overlap see https://github.com/TYPO3-Documentation/TYPO3CMS-Reference-CoreApi/issues/2921Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:111ReplyDoneFeature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:115ReplyDoneFeature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:127ReplyDoneFeature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130ReplyThe reason I started this patch: For TYPO3 v11: I was setting up testing for an extension and on composer install in the extension directory, the extension was not installed into the .Build/Web/typo3conf/ext folder. It did not have a Public/Resources folder. After adding the Public/Resources folder the extension was installed in .Build/Web/typo3conf/ext. I assumed this was related to the FOLLOWUP patch 71029 which also made changes to this changelog. https://review.typo3.org/c/Packages/TYPO3.CMS/+/71029/7/typo3/sysext/core/Classes/Composer/PackageArtifactBuilder.php#180 I hope this is correct, if not I would remove that point.Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:132ReplyDone -
core-ci voted on patch set 3 V +1 05/06/23, 15:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41630
-
Helmut Hummel commented on patch set 3 05/08/23, 07:22
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130ReplyOh, ok I see. This is a bit tricky to understand, I agree. TYPO3 in Composer mode since TYPO3 v11: 1. TYPO3 is able to recognizes extensions being installed in any directory, not only any of the ext folders 2. These directories are: the vendor folder and the root package (e.g. when testing extensions) 3. When these extensions only have private resources, everything will work as before, the PackageManager will resolve the correct packagePath to those extension. 4. When these extensions have public resources, by default, no URL can be generated, because these directories are not publicly exposed. 5. To ease the migration, the change you are referencing, is tweaking the location of the extension if it is the root package, to be exposed to typo3conf/ext, because this folder by default is exposed by the web server. If your extension however does not have or does not need public resources, there is no need to add them to activate this tweak -
Sybille Peters commented on patch set 3 05/08/23, 17:12
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130ReplyHelmut, your view is for running the extension in a TYPO3 installation. My view is (at least for the last point) setting up testing: > Extensions MUST have a :file:`Public/Resources` folder if they are to be installed in :file:`.Build/Web/typo3conf/ext` (or respective path) for testing. This is only relevant for a specific test setup but is quite commonly used. And in this case the extension MUST have a Resource/Public dir (even if otherwise it does not need it) since the change. For example, if you use v11 + typo3/testing-framework 6 and using a test setup in the extension with runTests.sh / docker-compose.yml with explicit paths, such as: Web/typo3conf/ext/styleguide/Build/phpunit/FunctionalTests.xml (which is commonly used) https://github.com/TYPO3/styleguide/blob/11/Build/testing-docker/docker-compose.yml git clone https://github.com/TYPO3/styleguide.git git checkout 11 rm -rf Resources/Public Build/Scripts/runTests.sh -s composerUpdate Build/Scripts/runTests.sh -s functional will fail. Could not read "Web/typo3conf/ext/styleguide/Build/FunctionalTests.xml". because .Build/Web/typo3conf/ext/styleguide does not exist. I had this problem recently with another extension. Creating a Resources/Public folder fixed it. -
Helmut Hummel commented on patch set 3 05/09/23, 06:19
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130Reply> your view is for running the extension in a TYPO3 installation. > My view is (at least for the last point) setting up testing: No, I am explicitly talking about testing and how TYPO3 is working in Composer mode > Build/Scripts/runTests.sh -s functional > Could not read "Web/typo3conf/ext/styleguide/Build/FunctionalTests.xml". If that is the case, this is a shortcoming of either the testing framework or the runTests.sh script. This should not be documented generally for TYPO3, but rather fixed in the testing framework or changed in that particular testing setup. There is no need for a Resources folder (and the therefore created symlink to typo3conf/ext), when the extension otherwise does not need or have public resources. Testing framework and testing setup (docker/ runScript.sh) should be adapted accordingly. Afaik, the testing framework in current version can work without such symlink in typo3conf/ext -
Stefan Bürk commented on patch set 3 05/11/23, 19:23
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130ReplyTF6 supports TYPO3 v10 and v11. This version had a composer script to ensure symlinks. For TYPO3 v11 it does nothing and let the core handle the symlinking stuff. TF7 is for TYPO3 v11 + v12, and the symlinking or not symlinking stuff is handled by typo3/cms-composer-installers + core in some wired connections. If the "root composer.json" is of type "typo3-cms-extension" AND there is no public resouce available, TYPO3 v11 is not creating a symlink for the root extension to the typo3conf/ext folder. Even with `cms-installers ^3`. Reading the core file and comment, this is not a bug but working as `implemented`: > For root packages of other types or extensions without public resources, no symlink is created and the package path stays to be the composer root path. See: https://github.com/TYPO3/typo3/blob/25f890ad88d26430658b60701d01054271bf93ce/typo3/sysext/core/Classes/Composer/PackageArtifactBuilder.php#L228-L229 So, something was started but not finished there already. I can only guess, but this may have been a first step towards the changed behaviour for composer mode started with cms-composer-installers 4RC1 + 5 to no longer providing whole extensions in typo3conf/ext/ for composer mode and only symlinking. TF6 and TF7 back hard required extensions to be in typo3conf/ext/ and system extensions in typo3/sysext/ ... in all cases. So, yet another "work" which have been begun, enforced with `cms-composer-installers` 5 in TYPO3 v12 - but not finished. And left extension/project testing broken. Currently, for building "functional test instances", TF can now deal with these cases for symlinking INTO the test instance correctly for TYPO3 v11/v12 and cms-composer-installers 3/4RC1/5 with TF 7.0.2 and TF 8.0.1. No one is now handling the case, that if the root page is a extension, that the extension is symlinked to the instance folder created by a composer install, e.g. into `.Build/` for TYPO3 v11 with cms-composer-installers v3 (4RC1 is okay to leave the extension out and no public resources means no assets symlink - working as intended I would say). Backporting the stuff to TF6 failed then for working with TYPO3 v10, therefore I avoided that part. This is leavign it to a not correct symlinked state with "TYPO3 v11, cms-composer-installers ^3 and TF 6.x". Sybille "tried" to document this gap, which is expressivly "intented behaviour" reading the h `handleRootPackage()` method docblock. (can be mitigated with "simply" symlink in composer install in the extension root composer.json - if people knows this, which are not a lot of people". Beside functional testing, this would also not install the "root extension" into the "build instance" into ".Build/Web" ( or which path is configured in the root composer.json. I'm not that much into the "v11" core code. Can you suggest a solution to fix it "there" for this "scenario" ? than a docuemnting would be "obsolete". For the constellation "TF6.x + TYPO3 v11 + cms-composer-installers 4RC1" my "bridge plugin" can still be used doing that job (if people know about this. When switching to TF7 it's replaced by the TF. This would leve the "build instance" broken, but tests working. More headache .... (Personally not using the ".Build/" instance as instance on extension development, but some developers may be doing that. But if I got it right, it was a planed move to let core + installers handle the correct symlinking etc and left the gap. Which leads us back to the V11 core handleRootPackage()" part. @Helmut do you have a suggestion how to fix it there in v11 to close this "gap" ? My head is running in cicles now ;) ;) -
Stefan Bürk commented on patch set 3 05/11/23, 20:01
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130ReplyTF7.x V11 + installers ^3 could change the "test invocation path" if they used `.Build/Web/typo3conf/ext/<rootextkey>/Tests/Functional/...` .. rest is handled by the TF now. And a extension without public resources do not need to be in ".Build/Web/typo3conf/ext/<rootextkey>" it the instance is used as instance. And if there are any, it would be symlinked. That was not possbile with TF7 until my changes for symlinking into TF7. Bridge plugin "mitigated" that also, which now enforces the force path change already for installers 3. My bridge plugin only kicks in for installers 4RC1 + 5 (without TF7+ installed). That means, that dual core testing with TF6.x needs to use dynamic entry paths for testing regarding core versions. or a semi revert for v11 for root extension pacakge type ??? What would be your suggestion Helmut ? -
Helmut Hummel commented on patch set 3 05/11/23, 21:54
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130Reply> What would be your suggestion Helmut ? Since you seen to know better what needs to be done best, go ahead -
Helmut Hummel commented on patch set 3 05/11/23, 21:55
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130ReplyDone -
Helmut Hummel commented on patch set 3 05/12/23, 01:57
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130ReplyOk, more constructive mode again. 1. We are discussing adding more information to TYPO3 11 changelog to help clarify the impact of replacing the PackageStates.php file when installing TYPO3 with Composer. 2. There two totally different concerns to consider regarding the impact a) Impact on TYPO3 11 installations (which this document is almost exclusively about currently) b) Impact on testing extensions and/ or projects It for sure is worth documenting both. If we are doing both, though, my suggestion would be to clearly separate these concerns. If we mix these concerns, I fear it will cause more confusion than it helps for clarification. 3. Those concerns also have its place in code. How TYPO3 installations work should be handled in TYPO3 code. How Testing TYPO3 installations and extensions work, should be handled in the testing framework. 4. When doing a "composer install" of a root package, that has a requirement to typo3/cms-core, the only concern of Composer and the TYPO3 Composer plugin, is to ensure to put everything in place, so that the TYPO3 CMS application can run. In that sense, the only goal of the here cited "handleRootPackage" method, is to ensure, no matter what kind of package the root package is, a composer install will lead to state, that the TYPO3 application can run successfully. This code, like basically all other code in the core repo (except the tests and build folder) has zero purpose to do anything for testing. The intention is like so: When the root package is an extension, TYPO3 will register the root package with its correct path as extension, despite not being located in the typo3conf/ext folder. This marks the first step of allowing TYPO3 to work with extensions packages, that are not located in one of those (now legacy folders). However, the root package in a composer installation always is OUTSIDE the webserver's document root. This would have meant, that such extensions would not be able to provide public assets. To mitigate that fact, the root package is linked to the typo3conf/ext folder, which typically is within the public folder, alas the document root. This was implemented BEFORE TYPO3 with composer installer 4 was able to publish public assets from extensions that are installed outside the document root. For the PackageManager of a TYPO3 installation, all that is handled transparently. There is no difference whether an extension is required by another package, or serves a root package, everything is working exactly the same. The information about the directories where each extension of such installations reside is now in exactly one place: the PackageArtifact.php file in the vendor folder. It is the only reliable source of truth, everything else (like the symlink to typo3conf/ext) is an implementation detail. 5. Since composer installers 4 this behaviour (symlinking to typo3conf/ext) is obsolete and therefore disabled in the code. The PackageArtifact.php however remains the reliable source of truth. 6. There is a lot of unfinished work regarding this concern. A few bugs in edge cases and a bunch of missing features, to make everything more polished. But no critical blockers. 7. Testing is a different concern. Quite a lot of unfinished work was left over in that are since making composer installer 5 required for TYPO3 12.0. With the latest releases of the testing framework 7.0.2, everything is in a pretty good shape though. Stefan worked on making an important change how it works, that is not relying on hard coded directory structures, but using the actual information from Composer where the packages are installed. With this, there is no need to create any resources folder. 8. Back to the topic. Since this change targets TYPO3 11, 12, 13 and, as outlined, everything is in place for these versions, I don't see the need to document something (the need to create a Resources folder), which is needed due to an (unfortunate) shortcoming in older testing framework versions. last but not least. If there is the wish to document a migration path for testing here, I'm fine with it. I suggest to create a dedicated headline for this topic and maybe straightforward suggest to to use TF 7.x for testing TYPO3 11 projects (instead of 6.x) AFAIK there also is documentation on testing. It may be worth updating that as well accordingly and maybe reference that documentation from here as well. -
Helmut Hummel wrote 05/12/23, 01:57
Removed cc Helmut Hummel. -
Helmut Hummel commented on patch set 3 05/12/23, 02:00
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:130Replylast thing: despite being generally busy, I never refused to have a talk when being asked for. So when it helps to clarify more things during a talk about all this, feel contact me. -
Sybille Peters commented on patch set 3 05/12/23, 08:14
@Helmut @Stefan My focus here: What can we do here, right now about: in v11, the extension will not be installed, if no Resources/Public folder exists and composer install / update is called in the extension folder. It will not be installed (or symlinked) to .Build/Web/typo3conf/ext and also not installed to .Build/vendor. This DOES happen in v10. In v11 it only happens if Resources/Public folder exists. In v12 it does not happen at all. This is actually a pretty minor "edge case". It would be a pity if we can't solve this now (e.g. by adding a half sentence to a changelog). Questions: 1. Should we document this specific thing for v11 in this changelog (since the behaviour changed and it might affect extension authors)? Alternative could be, we document it here: https://docs.typo3.org/m/typo3/reference-coreapi/11.5/en-us/Testing/ExtensionTesting.html 2. If yes, what do we give as advice? a) create a Resources/Public folder if extension testing depends on the extension to be installed into .Build/Web/typo3conf/ext (runTests.sh setup usually depends on it but there may be other use cases) b) create the symlink yourself for v11 only c) change the paths in the scripts (e.g. Build/testing-docker/docker-compose.yml (It is different in main, but it may not be possible to do it exactly as in main and it seems complicated) d) just mention this and give no advice e) ? "runTests & friends" is - I would say - semi-official. It can be copied and modified by extension authors. And it is explained in official docs. It is NOT supplied to extension authors as official package. So we have a dilemma - with this "semi-official" 'ness - we cannot really require for styleguide to change the runTests etc. script so they MIGHT also work with extensions in v11 which do not have Resources/Public folder. We already have the copied and modified versions "in the field". (styleguide itself does not have this problem as it has a Resources/Public folder). (I have used the "styleguide" as example because it seems to me the most up-to-date and best maintained though the official docs point to enetcache, see https://docs.typo3.org/m/typo3/reference-coreapi/11.5/en-us/Testing/ExtensionTesting.html). I think it would be good if we had an official runTests etc. scripts for extensions. So we say - ok this is the "master" (or "blueprint") script for extension testing. And this is where it is documented. It does not have to be the core team which maintains this. But, that is also something for the future we can't solve immediately, and certainly not for v11. Additional general discussion should maybe be done elsewhere. -
Helmut Hummel commented on patch set 3 05/12/23, 08:35
Reply
> in v11, the extension will not be installed, if no Resources/Public folder exists and composer install / update is called in the extension folder. It will not be installed (or symlinked) to .Build/Web/typo3conf/ext and also not installed to .Build/vendor. This DOES happen in v10. In v11 it only happens if Resources/Public folder exists. In v12 it does not happen at all. I understand. The details in this very topic are quite important to me personally, hence the discussion here. Therefore I try to make another approach in explaining: Symlinking an extension to typo3conf/ext for functional testing purposes was (IRC) something I have done as a HACK, to trick TYPO3 and the testing framework(s) to recognize the extension as part of TYPO3. For me this was never intended to be a permanent solution. I did not communicate it as such, but it worked well enough so that this hack was at some point promoted to be the standard way of how to solve things, thus was introduced as part of the testing framework at some point. The necessity of hack has technically become almost obsolete with TYPO3 v11. The only thing I am suggesting here is to NOT document the workings of a hack for TYPO3 v11 and higher, when this hack in fact became obsolete with this exact version. What I am suggesting instead, is highlighting that it is a hack and what can/ must be done to not rely on it for TYPO3 v11 (and higher) extension testing. In that regard, documenting to create a Resources/Public folder, is documenting how to use another hack (creating a not required folder) to a hack and from my perspective therefore highly misleading. -
Sybille Peters uploaded patch set 4
V +105/12/23, 17:26 -
Sybille Peters updated the commit message in patch set 5 05/12/23, 17:36
-
core-ci voted on patch set 4 V +1 05/12/23, 17:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41780
-
core-ci voted on patch set 5 V +1 05/12/23, 17:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41782
-
Sybille Peters commented on patch set 5 CR -1 06/16/23, 06:00
I will abandon now. It was a bit awkward from the start - adding a Migration section to a Feature changelog. The motivation was one edge case. We had multiple discussions, I was on holiday and to be honest the entire topic is currently not clear enough for me to keep working on this. In any case, probably better to update the main docs if necessary. -
Sybille Peters abandoned the change 06/16/23, 06:00
See comment.
Feature-94996-ConsiderAllComposerInstalledExtensionsAsActive.rst:104