Review main Merged

91316[BUGFIX] Prevent CLI cleanup:localprocessedfiles cleaning all files

Česlav Przywara CR +2 V +2 Easy +8 −11 updated 10/30/25, 07:25

Commit message

The CLI command "bin/typo3 cleanup:processedfiles" allows to
clean up resized / cropped / converted variations of media files,
by removing generated files which are no longer actively
referenced.

With #106002 a change was introduced that optimized directory
and file lookups with their "path generation". This lead to
duplicated "//" parts in the path that the cleanup command
uses to match referenced files with existing files.

Functional tests for this CLI command did not catch this
change in behaviour, because with the addition of the
"--all" option in #102582 those tests made wrong directory
assumptions, leading to ineffective checks to verify
only referenced files are removed.

This patch removes duplicate "//" path segments. This will
make the check for referenced files succeed again, and no
longer remove processed files which were still in use.
Previously, the presence of such segments in a path had been masked
(resolved) by using `SplFileInfo::getRealPath()`, which was
replaced in #106002 with `SplFileInfo::getPathname()`. See
[1] for more details on the variables

(Note that "processed files" can be regenerated like cache files,
when absent. This is a performance intensive operation though,
and not wanted when using the CLI command in a scheduler task
for example)

This patch also fixes the expectations and fixtures in the
related functional tests.

A duplicate call to SplFileInfo::isFile() is also removed by
this patch, as this verification is already executed
in the main loop of `getFilesOfFolderRecursive`.

[1] Details

Variables would for example contain:

- `$basePath` =  `/var/www/html/public/fileadmin/somethingelse/`
- `$folder->getIdentifier()` = '/_processed_/';

By using "ltrim()" here on the folder identifier, the leading
slash is removed and returns:

`/var/www/html/public/fileadmin/somethingelse/_processed_/`
  instead of:
`/var/www/html/public/fileadmin/somethingelse//_processed_/`

which would later be stripped to

`fileadmin/somethingelse/_processed_/...`
  instead of:
`fileadmin/somethingelse//_processed_/...`

Review

  • Code review CR +2 Done
    • Helmut Hummel +2
    • Garvin Hicking +1
  • Verified V +2 Done
    • Helmut Hummel +2
    • Garvin Hicking +1
    • core-ci +1 CI

Issues 4

Files 4

Review log 1724

  1. Česlav Przywara uploaded patch set 1 10/28/25, 17:23

  2. core-ci voted on patch set 1 V +1 10/28/25, 17:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/87869

  3. Garvin Hicking commented on patch set 1 10/28/25, 19:23

    we should dig deeper here, fixing the fixture like this patxh without any failures is definitely a bad smell, we should rather fix the test including the fixture - or use completely obvious faked paths if they don't matter do not give the impression the paths are properly built and evaluated
  4. Česlav Przywara commented on patch set 1 10/29/25, 09:32

    Reply

    I too expected some failure to pop up... I'll try to spin up the functional tests locally and see whether I can wrap my head around the test and then maybe come with some improvements.
  5. Česlav Przywara uploaded patch set 2 V +1 10/29/25, 10:22

  6. core-ci voted on patch set 2 V -1 10/29/25, 10:40 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/87934

  7. Česlav Przywara commented on patch set 2 10/29/25, 10:47

    Reply

    I found out that not only processed image path was incorrect in fixture data, but also storage UID. I fixed both, updated fixture data for databaseRecordForMissingFileIsDeleted test and also updated related asserts in two other test methods. The functional tests suite fails now, but the failure is due to the issue #107864 that got me investigating the test in the first place. Patch incoming.
  8. Česlav Przywara uploaded patch set 3 V -1 10/29/25, 11:07

  9. core-ci voted on patch set 3 V +1 10/29/25, 11:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/87946

  10. Garvin Hicking updated the commit message in patch set 4 V +1 10/29/25, 13:14

  11. Garvin Hicking updated the commit message in patch set 5 10/29/25, 13:16

  12. Garvin Hicking commented on patch set 5 10/29/25, 13:18

    sys_file_processedfile.csv:4

    What about this path, this is also having an "invalid" path? Will need to wrap my head around this, but thanks already for investigating and proposing a patch! Since the tests were introduced with my "--all" patch I really need to figure this out and remember :-)
  13. core-ci voted on patch set 4 V +1 10/29/25, 13:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/87967

  14. core-ci voted on patch set 5 V +1 10/29/25, 13:34 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/87969

  15. Česlav Przywara commented on patch set 5 10/29/25, 14:56

    sys_file_processedfile.csv:4 Reply

    This entry has an "invalid" path on purpose. As the filename suggests, it refers to non-existing file. The databaseRecordForMissingFileIsDeleted test runs the clean-up command without --all option, so only entries for which there are no processed files found (ie. with missing files like this one) should be deleted from database. In the databaseRecordForMissingFileIsDeleted test, database state is compared against oneDeleted.csv fixture - as the filename suggests, one entry should be deleted in comparison to sys_file_processed.csv fixture. This wasn't the case before this patch, because the two "ReferencedImage" entries has been deleted as well due to invalid storage UID and path. That is why after fixing storage UID and path I had added both entries to oneDeleted.csv as well. I hope this is clearer now. The tests as such are well thought-out, the problem was likely introduced due copy-paste with haste :-)
  16. Garvin Hicking updated the commit message in patch set 6 V +1 10/29/25, 19:43

  17. Garvin Hicking commented on patch set 6 CR +1 V +1 10/29/25, 19:45

    sys_file_processedfile.csv:4 Reply

    You are great! That helped me so much and I could finally find a few minutes to really go through it all. This works well. I hope you don't mind that I rephrased the commit message a bit with details for other reviewers and writing helps me a bit to process things.
  18. Stefan Bürk commented on patch set 6 10/29/25, 19:54

    sys_file_processedfile.csv:4 Reply

    Maybe we should add a `#` comment above the line to explicitly state that the "path" is "invalid" by intention (explaining it). Just to avoid that at some one some what to update this to a valid one.
  19. core-ci voted on patch set 6 V +1 10/29/25, 19:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/88037

  20. Garvin Hicking commented on patch set 7 CR +1 V +1 10/29/25, 20:51

    Published edit on patch set 6.
  21. Garvin Hicking commented on patch set 7 CR +1 V +1 10/29/25, 20:51

    sys_file_processedfile.csv:4 Reply

    Yeah sure, good idea!
  22. core-ci voted on patch set 7 V +1 10/29/25, 21:02 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/88049

  23. Helmut Hummel voted on patch set 7 CR +2 V +2 10/30/25, 07:16

  24. Helmut Hummel merged patch set 8 10/30/25, 07:25