#100155Parallel processing of image results in missing processed file entry
Description
We encountered the following situation in one of our projects:
Some images sometimes are delivered as is, instead of processed by TYPO3. There is an entry within table sys_file_processedfile that has no name, and no width and height:
SELECT from_unixtime(sys_file_processedfile.tstamp) as 'created',
sys_file_processedfile.storage as 'processed storage uid',
processedStorage.name AS 'processed storage',
sys_file.storage as 'sys file storage uid',
fileStorage.name AS 'sys file storage',
sys_file.identifier,
sys_file.uid,
sys_file_processedfile.identifier,
sys_file_processedfile.name,
sys_file_processedfile.width,
sys_file_processedfile.height
FROM `sys_file_processedfile`
LEFT JOIN sys_file ON sys_file_processedfile.original = sys_file.uid
LEFT JOIN sys_file_storage as fileStorage ON sys_file.storage = fileStorage.uid
LEFT JOIN sys_file_storage as processedStorage ON sys_file_processedfile.storage = processedStorage.uid
WHERE (sys_file_processedfile.width = 0 OR sys_file_processedfile.height = 0 OR sys_file_processedfile.name IS NULL)
We patched some logging into image handling with the following result:
Tue, 28 Feb 2023 11:42:27 +0000 [DEBUG] request="23d00f57c5b3e" Execute imagemagick command. - {"cmd":"'/usr/bin/gm' 'convert' -interlace None -auto-orient +profile '*' -auto-orient -geometry 1280x2522! -colorspace RGB -quality 80 -crop 1280x2522+0+0! +repage -colorspace RGB 'jpg:/var/www/project_name/release_2023-02-21T092248Z/web/fileadmin/user_upload/something-_Topptur89784654.jpg[0]' '/var/www/project_name/release_2023-02-21T092248Z/web/typo3temp/assets/images/74228ce61503696e378a5f293180a325.jpg'"}
Tue, 28 Feb 2023 11:42:27 +0000 [DEBUG] request="ddb113bc8c539" Execute imagemagick command. - {"cmd":"'/usr/bin/gm' 'convert' -interlace None -auto-orient +profile '*' -auto-orient -geometry 1280x2522! -colorspace RGB -quality 80 -crop 1280x2522+0+0! +repage -colorspace RGB 'jpg:/var/www/project_name/release_2023-02-21T092248Z/web/fileadmin/user_upload/something-_Topptur89784654.jpg[0]' '/var/www/project_name/release_2023-02-21T092248Z/web/typo3temp/assets/images/74228ce61503696e378a5f293180a325.jpg'"}
Tue, 28 Feb 2023 11:42:27 +0000 [DEBUG] request="23d00f57c5b3e" Executed imagemagick command. - {"ret":"","cmdOutput":[],"cmdReturn":0}
Tue, 28 Feb 2023 11:42:27 +0000 [DEBUG] request="23d00f57c5b3e" Cache image dimensions. - {"returnArr":[1280,2522,"jpg","/var/www/project_name/release_2023-02-21T092248Z/web/typo3temp/assets/images/74228ce61503696e378a5f293180a325.jpg"]}
Tue, 28 Feb 2023 11:42:27 +0000 [DEBUG] request="ddb113bc8c539" Executed imagemagick command. - {"ret":"","cmdOutput":[],"cmdReturn":0}
Tue, 28 Feb 2023 11:42:27 +0000 [DEBUG] request="ddb113bc8c539" Output file did not exist after executing imagemagick. - {"outputFile":"/var/www/project_name/release_2023-02-21T092248Z/web/typo3temp/assets/images/74228ce61503696e378a5f293180a325.jpg"}
Tue, 28 Feb 2023 11:42:27 +0000 [DEBUG] request="ddb113bc8c539" Keep original file as processed file. - {"targetFile":"/user_upload/something-_Topptur89784654.jpg","sourceFile":"/user_upload/something-_Topptur89784654.jpg","localFile":null}
As you can see there are two concurrent requests at the same time. Both requesting the same source image with same configuration. It looks like the first passes, while it removes the intermediate result leading to an error in the second request. The result of the 2nd request is then stored for recurring requests.
Page requests are locked, but maybe this is due to requesting same page in different languages, or the same image is used with same configuration on a different page?
Slack message: https://typo3.slack.com/archives/C03AM9R17/p1678717091808029
Reviews 1
Fix this issue
Start the commit message of your patch with this. Add what the patch does, and the versions it goes to in "Releases:". The changes above were merged or abandoned: read them before you start. How to create a patch
- Commit message
-
[BUGFIX] Parallel processing of image results in missing processed file entry Resolves: #100155 Releases: main
Related issues 1
History 46
-
Gerrit Code Review pushed patch set 1 for main to change 78115 status
NewUnder Review <span data-processor="relativetime" data-value="2023-03-14T07:42:01+00:00" > 03/14/23, 07:42 </span> -
Gerrit Code Review pushed patch set 2 for main to change 78115 <span data-processor="relativetime" data-value="2023-05-03T09:35:02+00:00" > 05/03/23, 09:35 </span>
-
Daniel Siepmann commented and changed removed the assignee <span data-processor="relativetime" data-value="2023-05-08T06:51:17+00:00" > 05/08/23, 06:51 </span>
-
Rémy DANIEL changed Related to #98700 <span data-processor="relativetime" data-value="2026-02-20T15:27:30+00:00" > 02/20/26, 15:27 </span>
-
Rémy DANIEL commented <span data-processor="relativetime" data-value="2026-02-20T15:30:07+00:00" > 02/20/26, 15:30 </span>
The image processing use the locking mecanism in order to not process the same image twice.
This mecanism has been optimized by #98700 -
Benni Mack changed status
Under ReviewAccepted <span data-processor="relativetime" data-value="2026-11-09T10:27:36+00:00" > 11/09/26, 10:27 </span>
I tried to solve this with my knowledge in a time box defined by our customer, but did not succeed.
I therefore free this issue up for someone else to pick up.
Our workaround: We will add a small command that will remove the corrupted processed files entries. We will use that one as a scheduler task.