Review main Merged

89755[BUGFIX] Check correct file location in DeferredBackendImageProcessor

Markus Klein CR +2 V +2 Easy +1 −1 updated 07/25/25, 16:50

Commit message

The condition whether a task can be processed
by this processor involves a file existence check.
It should check, if the expected target file
for the processed file already exists within
the processed-files folder.

The check was wrong, because it used the
root processing folder instead of the nested
processing folder.

Review

  • Code review CR +2 Done
    • Garvin Hicking +2
    • Andreas Kienast +1
    • Simon Schaufelberger +1
  • Verified V +2 Done
    • Garvin Hicking +2
    • Andreas Kienast +1
    • Simon Schaufelberger +1
    • core-ci +1 CI

Issues 1

Files 1

Review log 2326

Show 1 earlier messages
  1. Markus Klein uploaded patch set 1 06/18/25, 12:17

  1. core-ci voted on patch set 1 V +1 06/18/25, 12:24 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/80462

  2. Garvin Hicking commented on patch set 1 06/24/25, 08:45

    I'm afraid there's something fishy going on here. I tried to follow the debugging trail, and at least in my instance (main) with a local file driver, using the "getIdentifier()" call leads to fileExists check (`LocalDriver->fileExists()`) on a location like: ``` /var/www/html/public/fileadmin/_processed_/_processed_/1/9/csm_Screenshot_2025-06-23_at_19.33.17_9e30a46142.png ``` whereas with "getTargetFilename" it checks on: ``` /var/www/html/public/fileadmin/_processed_/csm_Screenshot_2025-06-23_at_19.33.17_9e30a46142.png ``` Now for the fun part: BOTH files do not actually exist. My sys_file_processedfile has this: ``` INSERT INTO `sys_file_processedfile` (`uid`, `tstamp`, `crdate`, `storage`, `original`, `identifier`, `name`, `processing_url`, `configuration`, `configurationsha1`, `originalfilesha1`, `task_type`, `checksum`, `width`, `height`) VALUES ('1250', '1750749497', '1750749497', '1', '66796', '/_processed_/1/9/preview_Screenshot_2025-06-23_at_19.33.17_54039d96a5.png', 'preview_Screenshot_2025-06-23_at_19.33.17_54039d96a5.png', '', X'613A323A7B733A353A227769647468223B693A36343B733A363A22686569676874223B693A36343B7D', '551dfa8957f1a04693c61acf34bc959a1ca971c4', '4b7a33627da85f20798b42154d97043cd6d550b6', 'Image.Preview', '54039d96a5', '64', '26'), ('1251', '1750751054', '1750749504', '1', '66796', '/_processed_/1/9/csm_Screenshot_2025-06-23_at_19.33.17_9e30a46142.png', 'csm_Screenshot_2025-06-23_at_19.33.17_9e30a46142.png', '', X'613A323A7B733A353A227769647468223B733A333A22333263223B733A363A22686569676874223B733A333A22333263223B7D', '48ec22f851d7822181aeed9649f97929e5f0c410', '4b7a33627da85f20798b42154d97043cd6d550b6', 'Image.CropScaleMask', '9e30a46142', '32', '32'), ('1252', '1750749504', '1750749504', '1', '66796', '/_processed_/1/9/csm_Screenshot_2025-06-23_at_19.33.17_b499b5da65.png', 'csm_Screenshot_2025-06-23_at_19.33.17_b499b5da65.png', '', X'613A373A7B733A353A227769647468223B4E3B733A363A22686569676874223B4E3B733A383A226D696E5769647468223B4E3B733A393A226D696E486569676874223B4E3B733A383A226D61785769647468223B693A3136363B733A393A226D6178486569676874223B693A3131353B733A343A2263726F70223B4E3B7D', '42bbd46565440d020e85026168e5aeb9653d29fa', '4b7a33627da85f20798b42154d97043cd6d550b6', 'Image.CropScaleMask', 'b499b5da65', '166', '67'); ``` The only file that really exists on my fileadmin processed directory is `/_processed_/1/9/csm_Screenshot_2025-06-23_at_19.33.17_b499b5da65.png`. When that file is requested through the deferred backend processor, the sys_file_processedfile entry gets updated. Every time. Because the file never gets written (because the other existant file is re-used). The UID #1251 is the entry where `tstamp` gets bumped every time, which has the matching filename, which does not exist on the server itself. I am not sure if that is the intended behavior - but it produces thumbnails for a LocalDriver result. Changing the deferred backend processor like in this patch would actually still work with LocalDriver, because the file again would never exist in either case.
  3. Markus Klein commented on patch set 1 06/24/25, 09:15

    Reply

    Great finding, but a different story. The files do not exist, because they are deferred. They are created on demand. But this patch only is about the file path that will and must for sure never exist. "/_processed_/csm_..."
  4. Benjamin Franzke commented on patch set 1 06/24/25, 09:52

    Reply

    But Garvin is right, `$task->getTargetFileName()` delivers a composed filename – see https://github.com/TYPO3/typo3/blob/a27d0a77b1f24659e22fac66af61393be17e51f8/typo3/sysext/core/Classes/Resource/Processing/AbstractTask.php#L66 or even https://github.com/TYPO3/typo3/blob/a27d0a77b1f24659e22fac66af61393be17e51f8/typo3/sysext/core/Classes/Resource/Processing/ImageCropScaleMaskTask.php#L71 – an information which `$task->getTargetFile()` does not reflect (yet), because it seems to be meant to be set by the processors, as seen in the local image processor: https://github.com/TYPO3/typo3/blob/a27d0a77b1f24659e22fac66af61393be17e51f8/typo3/sysext/core/Classes/Resource/Processing/LocalImageProcessor.php#L76 That means this check would use some kind of preliminary target file name, which is not actually used by the processors.
  5. Markus Klein commented on patch set 1 06/24/25, 11:36

    Reply

    Maybe I misunderstand you both, but: It does not matter which filename is used for this check, it is simply wrong. It has to use the full identifier, which includes the path. Processed files are always in sub-directories. All other places dealing with processed files are always using the identifier, correctly.
  6. Garvin Hicking commented on patch set 1 06/24/25, 12:16

    Reply

    But then the current patch would be incomplete? That file existance would be performed on a duplicate `_processed_/_processed_/..." structure, so we should have a solution in place that at least checks for a right directory, even though the rest of the processing seems buggy at this point. The way I read this, `$task->getSourceFile()->getStorage()->getProcessingFolder()->hasFile($task->getTargetFile()->getIdentifier());` currently always returns false, there is no situation where this returns true: Neither when a new image is created, nor when an existing thumb is deferredly rendered. You could use `!$task->getSourceFile()->getStorage()->getProcessingFolder()->hasFile('/path/to/hey-i-am-not-a-file.jpg');` and the code execution paths would be the same as they are now. If I understand right you want to fix this for a different driver than LocalDriver, right? So that driver doesn't return the composed filename?
  7. Markus Klein commented on patch set 1 06/24/25, 13:40

    Reply

    I did the major testing on the S3 driver. I did some testing on the local driver too. > currently always returns false, there is no situation where this returns true If you have multiple servers this can very well be the case, that some server created this in the meantime. But for normal instances this will always be false. I fully agree, this is not optimal. We generally have the problem that a non-existing processed file currently causes 3x a file_exists check, which is crazy if you consider that each of those can be a request to some remote system. (DAM or S3 or whatever) This patch ensures we at least check for a file that might exist at some point, and not for something that can technically never exist. (saving us useless I/O calls, possibly remote) > If I understand right you want to fix this for a different driver than LocalDriver, right? So that driver doesn't return the composed filename? This has nothing to do with the driver. This all happens just within FAL. The target file name is calculated within FAL. As Benjamin linked above.
  8. Garvin Hicking commented on patch set 1 06/24/25, 14:14

    Reply

    > This patch ensures we at least check for a file that might exist at some point, > and not for something that can technically never exist. (saving us useless I/O > calls, possibly remote) I don't want to be a stickler, but if we change it like in the patch, due to the duplication of `processed`, at least for LocalDriver exactly this will happen: it will check for a file that can never exist. Of course we could argue that since its broken anyways, and it fixes the scenario for storages where it hurts to have this lookup, we could live with it as a hotfix. I don't feel authoritative to make a decission for this, though. I think at best we somehow need to be able to let the Driver have a say in this? So we might need another interface method that allows the driver to tell the FAL what the deferred processing should check on...?
  9. Markus Klein commented on patch set 1 06/24/25, 20:28

    Reply

    > "duplication of processed" That's weird, I didn't have that. This is of course not the idea! I'll retest with LocalDriver again
  10. Markus Klein commented on patch set 1 07/04/25, 19:37

    Reply

    I did some more debugging with local storage. tl;dr: totally weird stuff going on. 1.) I prepared a folder with only a single image 2.) I used tile view 3.) I flushed all processed images via Install Tool (separate browser tab) 4.) I set a debug breakpoint in `DeferredBackendImageProcessor::canProcessTask` 5.) I refreshed the folder Observed behavior for `$task->getTargetFileName()` ("N" below) `$task->getTargetFile()->getIdentifier()` ("I" below) `$task->getSourceFile()->getStorage()->getProcessingFolder()->hasFile(...)` tested with N and I as argument ("RN" "RI" below): * First stop (while rendering the file list): I:'/Floral.jpg', N:'csm_Floral_c5d2dae158.jpg', RI:false, RN:false * Second stop (when loading the image via img-src): I:'/_processed_/f/8/csm_Floral_c5d2dae158.jpg', N:'csm_Floral_c5d2dae158.jpg', RI:false, RN:false Facts: * During first stop the check is totally useless with I and N, neither can possibly exist. * During second stop the check only makes sense with I, which could exist theoretically. * In both cases RI and RN are always false, naturally, since the processing hasn't happened yet. * The RI check could only be true if some other request created the file meanwhile, but didn't update FAL yet. (really rare case) * The RN check is completely pointless. * Once the second stop has happened, the DeferredBackendImageProcessor is never reached again on reloads of the file list, since the processed images are now present. Conclusion: * This patch makes sense as at least avoids one totally pointless file_exists check during the second stop. * The whole "hasFile" check is very questionable as it practically always contributes `true` to the condition (due to the negation). * When file_exists checks are expensive, this is a massive improvement performance-wise.
  11. Markus Klein commented on patch set 1 07/04/25, 19:53

    Reply

    Of course the patch needs to be updated to avoid the duplication of "_processed_" as noted above! Will take care. Unless we decide to kick the condition completely.
  12. Markus Klein commented on patch set 1 07/04/25, 19:56

    Reply

    And looks like we have the same issue in LocalImageProcessor.php:112 too ``` // explicitly check for the raw filename here, as we check for files that existed before we even started // processing, i.e. that were processed earlier if ($processingFolder->hasFile($task->getTargetFileName())) { ```
  13. Markus Klein commented on patch set 1 07/04/25, 20:02

    Reply

    `SvgImageProcessor` suffers too. I traced it back in history: During older times (I have a v6.2 lying around) this check was correct, as all files were stored directly in the processed-folder without a subfolder structure
  14. Markus Klein commented on patch set 1 07/04/25, 20:40

    Reply

    And here we have the culprit: https://review.typo3.org/c/Packages/TYPO3.CMS/+/36523 It was forgotten do adjust all these conditions.
  15. Markus Klein uploaded patch set 2 V +1 07/04/25, 20:49

  16. Markus Klein uploaded patch set 3 07/04/25, 20:50

  17. Markus Klein commented on patch set 3 07/04/25, 20:51

    Reply

    After some more digging. The other conditions are correct, as the only relevant thing is the call to `getProcessingFolder()` which has dual-functionality. This was wrong in the DeferredBackendImageProcessor. I updated the patch.
  18. core-ci voted on patch set 2 V +1 07/04/25, 20:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/81324

  19. core-ci voted on patch set 3 V +1 07/04/25, 20:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/81325

  20. Garvin Hicking commented on patch set 3 CR +1 V +1 07/07/25, 09:14

    Voting on this because the file checks at least use the proper API now (in my opinion). The problem that this still checks for a file that will never be found because of the whole complex workflow is something that we could/should fix in a follow-up. The way I understood is that this applies to the "local" FAL usage, but for other FAL storages like S3 this file existence check eases the burden of looks, so I'd say a "hotfix" is warranted until the real solution can be provided.
  21. Andreas Kienast commented on patch set 3 CR +1 V +1 07/25/25, 08:07

    I've verified this by using xdebug. Indeed, without the patch, wrong fiel paths are checked that never exist.
  22. Markus Klein commented on patch set 3 07/25/25, 08:48

    Reply

    Done
  23. Simon Schaufelberger voted on patch set 3 CR +1 V +1 07/25/25, 09:48

  24. Garvin Hicking voted on patch set 3 CR +2 V +2 07/25/25, 16:30

  25. Garvin Hicking merged patch set 4 07/25/25, 16:50