Review main Abandoned

89376[BUGFIX] Avoid exceeding `sys_registry.entry_key` length

Kevin Appelt CR 0 V 0 Normal +86 −6 updated 05/14/25, 07:20

Commit message

The `PathUtility::stripPathSitePrefix()` method has
been fixed with #106605 to only strip public path,
doing what the phpdoc block states to do.

The method has been used in several places getting
extension (package) initialization data from more
than one source, for example:

* `ext_tables_static+adt.sql` static database data
  files.
* Extension files `Initialisation/data.t3d` and/or
  `Initialisation/data.xml`.
* Site configuration from `Initialisation/Site/*/`.

Based on the nature of #106605 paths in deployments
or instances in composer mode it no longer removed
the similar prefix based on the length for `vendor`
instead of `public` and returning the full absolute
paths. Depending on the system and filesystem depth
this leads to path lengths greater than `sys_registry`
supports as `entry_key`, used within the data import
handling to track already import state, honoring it
with database error:

> An exception occurred while executing a query:
> Data too long for column 'entry_key' at row 1

This change introduces the dedicated internal
method `PathUtility::stripMultiplePathsPrefix()`
and replaces `stripPathSitePrefix()` in data
initialization/import related methods to restore
the old behaviour for now and mitigate the introduced
regression.

Replacing the registry cache keys with relative
paths to extension base using `EXT:path/file.txt`
is a reasonable way to streamline this even more,
but requires registry cache key migration handling
and adoption in quite some places and is considered
to be done as a dedicated TYPO3 v14-only change to
avoid introducing more regressions instead of fixing
one.

Review

  • Code review CR 0
  • Verified V 0

Issues 2

Files 6

Review log 2231

Show 56 earlier messages
  1. Kevin Appelt uploaded patch set 1 05/13/25, 15:14

  2. Stefan Bürk voted on patch set 1 CR -2 05/13/25, 15:16

  3. Stefan Bürk commented on patch set 1 05/13/25, 15:17

    That's not the solution as the method does not do what the comment says, and did before due to a bug !
  4. core-ci voted on patch set 1 V +1 05/13/25, 15:20 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78857

  5. Benni Mack commented on patch set 1 05/13/25, 15:57

    I think we should add a 2nd method to fall back to the EXT:path. Additionally, we could consider storing "EXT:my_ext/ext_tables_static+adt.sql" (just like this) in the registry $extTablesStaticSqlFile = $event->getPackage()->getPackagePath() . 'ext_tables_static+adt.sql'; $extTablesStaticSqlRelFile = PathUtility::stripPathSitePrefix($extTablesStaticSqlFile); We should just use a different piece of code in this place instead of using "PathUtility::stripPathSitePrefix".
  6. Stefan Bürk uploaded patch set 2 V +1 05/13/25, 17:02

    Copied Votes: * Code-Review-2 (copy condition: "changekind:NO_CHANGE OR **is:MIN**")
  1. Stefan Bürk commented on patch set 2 CR -2 05/13/25, 17:02

    On it ...
  2. core-ci voted on patch set 2 V +1 05/13/25, 17:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78862

  3. Stefan Bürk uploaded patch set 3 V +1 05/13/25, 17:31

    Copied Votes: * Code-Review-2 (copy condition: "changekind:NO_CHANGE OR **is:MIN**")
  4. core-ci voted on patch set 3 V +1 05/13/25, 17:36 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78863

  5. Stefan Bürk uploaded patch set 4 V +1 05/13/25, 17:54

    Copied Votes: * Code-Review-2 (copy condition: "changekind:NO_CHANGE OR **is:MIN**")
  6. core-ci voted on patch set 4 V +1 05/13/25, 18:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78865

  7. Stefan Bürk uploaded patch set 5 V +1 05/13/25, 20:00

    Copied Votes: * Code-Review-2 (copy condition: "changekind:NO_CHANGE OR **is:MIN**")
  8. core-ci voted on patch set 5 V +1 05/13/25, 20:07 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78866

  9. Stefan Bürk uploaded patch set 6 V +1 05/13/25, 20:23

    Copied Votes: * Code-Review-2 (copy condition: "changekind:NO_CHANGE OR **is:MIN**")
  10. core-ci voted on patch set 6 V +1 05/13/25, 20:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78868

  11. Stefan Bürk commented on patch set 6 CR -2 05/13/25, 20:32

    Reply

    @benni@typo3.org We could consider to inline the new internal PathUtility method within the differen consumer classes as private methods, but went with a shared interna one to have tests only once for it. I like the idea with the `EXT:path/file` syntax for the registry keys, but it turned out that there are multiple data imports with several different registry keys where this would be also be done, and some places requires quite some work. I tried it and some of that could be seen in some intermediate changesets, some not as I dropped them working on and making that change to big. I'd say that using `stripPathSitePrefix()` in these data import handling event listerner stuff for package activation chain was correct untion TYPO3 v12 where extension has been moved out of `public/typo3conf/ext` and kept in the vendor folder - which was hidden by the pure strlen based stripping in the fixed method. From my side, I would say that the current provided solution is the only reasonable one to fix that - except reverting #106605.
  12. Stefan Bürk uploaded patch set 7 05/13/25, 20:34

    Copied Votes: * Code-Review-2 (copy condition: "**changekind:NO_CHANGE** OR **is:MIN**") * Verified+1 (copy condition: "**changekind:NO_CHANGE**")
  13. Stefan Bürk updated the commit message in patch set 8 V +1 05/13/25, 20:38

    Copied Votes: * Code-Review-2 (copy condition: "changekind:NO_CHANGE OR **is:MIN**")
  14. Stefan Bürk voted on patch set 8 CR 0 05/13/25, 20:39

  15. Stefan Bürk commented on patch set 8 05/13/25, 20:39

    Reply

    Done
  16. core-ci voted on patch set 7 V +1 05/13/25, 20:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78869

  17. core-ci voted on patch set 8 V +1 05/13/25, 20:47 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78871

  18. Oliver Hader commented on patch set 8 05/13/25, 21:22

    Unresolved

    Code-wise, I found these additional occurrences (I did not test, nor debug these source locations): * \TYPO3\CMS\Core\Imaging\ImageResource::createFromImageInfo - the usage in the core is fine (using public path) - however, since this is a public function, it might be used with other paths as well in extensions * \TYPO3\CMS\Core\Package\PackageActivationService::reloadExtensionData - seems like this might produce incorrect paths * \TYPO3\CMS\Core\Utility\DebugUtility::debugTrail - seems like this might show incorrect paths * \TYPO3\CMS\Lowlevel\Command\CleanUpLocalProcessedFilesCommand::deleteFile - seems like this might produce incorrect paths * \TYPO3\CMS\Lowlevel\Service\CleanUpLocalProcessedFilesService::getFilesToClean - seems like this might produce incorrect paths
  19. Oliver Hader commented on patch set 8 CR -1 05/13/25, 22:00

    PathUtility.php:455

    The scope of this function is not clear... e.g. when having `/var/www/whatever/vendor/typo3/cms-core/` it would strip the prefix and the result would be `cms-core`. Using this result in any further processing is not deterministic. The actual use-case should be reflected by the function name, e.g. `stripPathForDebugLoggin` or `stripPathForProcessingInProjectRoot`.
  20. Stefan Bürk commented on patch set 8 CR -2 05/14/25, 00:28

    PathUtility.php:455 Reply

    Let's simply revert #106605 (tomorrow) to the old broken state, which has worked for all these places more or less without being notices over years. And think about a proper handling for dedicated use-cases with methods/handlings/tests and replace at corresponding places afterwards/over-time. Just for completness, taking your example `/var/www/whatever/vendor/typo3/cms-core/` with a public path of `/var/www/whatever/public` with the original code before #106605 returned `typo3/cms-core/`. Simple because it stripped the length of `/var/www/whatever/public` from `/var/www/whatever/vendor/typo3/cms-core/`. More over .. taken the same public path from above but providing a completly other path stripped it in a weired way (the original code of stripPathSitePrefix): ``` PUBLIC: /var/www/whatever/public $path.: /some/completly/unrelated/path RESULT: related/ath ``` Originally the phpdoc block for `stripPathSitePrefix` was ``` Strip first part of a path, equal to the length of public web path including trailing slash ``´ what described what has been done - based on the length of the public path. That has been rephrased to ` Strips the public web path from the provided $path ...` and ensuring that only a matching prefix of public path is removed. I'd say we need one or more precise methods for stripping or checking and investigate each usage what is need ... in a dedicated change and cover that wit tests .. and what makes sense for that usage or not. All of this returns to a huge change and long-time discussion and the reason why I now tend to simply revert to the orignal state before the "bugfix" and the "weired broken" state beeing "good enough". Beside that, the new method here would return for `/var/www/whatever/vendor/typo3/cms-core/` not `cms-core` but `typo3/cms-core/` when Environment::getProjectPath() is `/var/www/whatever` and `getPublicPath: /var/www/whatever/public`. Reason is the ordering of the candidates and the loop, which checks for the preix and only strips it when given in the $path ... and the next candidate works on the modifed (stripped) $path doing the check again and only stripping the candidate when $path starts with it, which means or composer mode: 1. check with prefix `/var/www/whatever/public` => skip to next candidate 2. check with prefix `/var/www/whatever/vendor` => strip and set `$path = `typo3/cms-core` 3. check with prefix `/var/www/whatever`=> skipped as $path does not start with it The "fixed" `stripPathSitePrefix()` however would also simply strip `/var/www/whatever/123456/other/folder/` down to `other/folder/ what the new method not does (only specific prefix candidates only when existing. Let's revert that and handle that later or each place what is really required/needed/wanted .... albeit I still postulate that the old method worked weired and not really deterministic at all, but worked for most parts based on pure magic - let's restore that for now. Okay for you to revert the bugfix and drop this one here ?
  21. Stefan Bürk commented on patch set 8 05/14/25, 00:36

    PathUtility.php:455 Reply

    With composer * `conig/vendor-dir: some/other/vendor/folder` and * `extra.typo3/cms.web-dir: changed-docroot-folder` will fail with the old unfixed variant, the "fixed" variant and also with this approach. Therefore, we should make that overall as a epic I guess.
  22. Andreas Kienast commented on patch set 8 05/14/25, 05:45

    PathUtility.php:455 Reply Unresolved

    I had a patch once for this exact case but it was never finished: https://review.typo3.org/c/Packages/TYPO3.CMS/+/80547
  23. Markus Klein updated the commit message in patch set 9 CR -1 V +1 05/14/25, 07:09

    Copied Votes: * Code-Review-2 (copy condition: "changekind:NO_CHANGE OR **is:MIN**")
  24. core-ci voted on patch set 9 V +1 05/14/25, 07:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/78887

  25. Stefan Bürk abandoned the change 05/14/25, 07:20

    in favour of reverting the original bugfix making this obsolete, see: https://review.typo3.org/c/Packages/TYPO3.CMS/+/89381