93498[TASK] Speed up responsive image loading from db
Commit message
Review
-
Code review CR +2 Done
- Garvin Hicking +2
- Simon Schaufelberger +1
-
Verified V +2 Done
- Garvin Hicking +2
- Simon Schaufelberger +1
- core-ci +1 CI
Issues 1
Files 5
- typo3/sysext/core/Classes/Resource/Processing/FileDeletionAspect.php+5 −10
- typo3/sysext/core/Classes/Resource/ProcessedFileRepository.php3+131 −36
- typo3/sysext/core/Classes/ServiceProvider.php+1 −0
- typo3/sysext/lowlevel/Classes/Command/CleanUpLocalProcessedFilesCommand.php+1 −1
- typo3/sysext/lowlevel/Classes/Service/CleanUpLocalProcessedFilesService.php4+38 −52
Review log 2328
Show 23 earlier messages
-
Ayke Halder uploaded patch set 1 03/31/26, 16:59
-
core-ci voted on patch set 1 V +1 03/31/26, 17:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99298
-
Sven Liebert commented on patch set 1 CR +1 V +1 04/02/26, 09:24
I think we can backport that to v13 according to what @benni@typo3.org wrote in slack
-
Ayke Halder commented on patch set 1 04/02/26, 09:30
Reply
Commit message changed: * v13 * make clear that this speeds up loading from db - and not browsers loading the images. -
Ayke Halder updated the commit message in patch set 2
CR +1V +104/02/26, 09:32 -
core-ci voted on patch set 2 V +1 04/02/26, 09:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99462
-
Sven Liebert voted on patch set 2 CR +1 V +1 04/02/26, 09:54
-
Benni Mack updated the commit message in patch set 3
CR +1V +104/22/26, 06:44 -
core-ci voted on patch set 3 V +1 04/22/26, 11:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/100782
-
Christian Kuhn commented on patch set 3 06/11/26, 23:35
thanks for this work! patch is good in general: this is one of the rarer places where db results can be runtime-cached effectively and still be kept coherent, *if* we centralize all db operations into the repository. i adapted two further cases and cleaned up everything a bit. -
Christian Kuhn uploaded patch set 4
V +106/11/26, 23:38 -
core-ci voted on patch set 4 V +1 06/11/26, 23:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/103414
-
Christian Kuhn voted on patch set 4 CR +1 V +1 06/11/26, 23:49
-
Ayke Halder commented on patch set 4 06/12/26, 05:58
CleanUpLocalProcessedFilesService.php:206Is the logic change intentional? E.g. should the edge case `configuration['pathType'] = 0` be considered? To not change the current logic use elvis operator `?:` here. -
Ayke Halder commented on patch set 4 06/12/26, 06:01
CleanUpLocalProcessedFilesService.php:206Reply… `configuration['basePath'] = 0` … -
Garvin Hicking commented on patch set 4 V +1 06/12/26, 07:40
This works for me. Queries down from 11 to 1, same output. Seems solid, just one question and the issue Ayke raised and then I'm good to vote/merge 😊ProcessedFileRepository.php:186Maybe for main we can think about dropping this serialization; we could use json_encode here as it's a lot faster (3msec vs. 30msec on 100_000 iterations), and I would like to prevent utilizations of serialize() where unserialization is not needed. I recognize that the stored sha1 then would change and invalidate all create processed files, so that's a kind of breaking change. Thoughts (follow-up patch, of course)?CleanUpLocalProcessedFilesService.php:206ReplyI agree, we should use ?: here -
Ayke Halder commented on patch set 4 V -1 06/12/26, 09:51
One point that gives me a headache: Let's say we have 100.000 Users and we create some badge-grapic with the username on top. As all these 100.000 badges are based on the same image file, and only differ in image processing instructions, with this change all the 100.000 variations get loaded from db - even when only a single image is needed. Please proof me wrong on this - or declare it a super-rare edge case. -
Christian Kuhn commented on patch set 4 06/12/26, 10:22
ProcessedFileRepository.php:186Replyyes, we need to think about it, there are issues on it, too. separate change, though. -
Christian Kuhn commented on patch set 4 06/12/26, 10:25
Reply
imho super edgy, and probably still not *that* slow when it's one query. goal is fe performance with the patch, and that's what we should look at primarily. if this *really* turns out to be an issue, we'll reconsider to find a solution. -
Christian Kuhn uploaded patch set 5
CR +1V +106/12/26, 11:20 -
Christian Kuhn commented on patch set 5 CR +1 V +1 06/12/26, 11:21
CleanUpLocalProcessedFilesService.php:206ReplyDone -
core-ci voted on patch set 5 V +1 06/12/26, 11:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/103444
-
Garvin Hicking commented on patch set 5 CR +1 V +1 06/12/26, 12:38
Reply
I made a test and created 200k variants of one original file. With 14 rows it took 0.16 microseconds to query and populate cache. With these 200k rows it took 294 microseconds. That is a huge relative increase I agree, however in the broad terms of page assembly for this huge amount of rows, 0.2 seconds is not really that bad of an impact, especially when at least for frontend rendering that result is cached, and for extension / badge usage scenarios, creating such an overlay might also be subject to caching. I think with relevant scenarios, this is a larger win than hit for this edge-scenario, but I'm happy you brought this up and thought about this, I found that very interesting to check out. -
Ayke Halder commented on patch set 5 06/13/26, 15:37
Reply
@lolli@schwarzbu.ch @garvin@hick.ing Thanks for looking into this. So I also think it's safe to merge this now.ProcessedFileRepository.php:186Reply@garvin@hick.ing Have a look at: 93502: [TASK] Migrate processed file configuration storage from serialize() to JSON | https://review.typo3.org/c/Packages/TYPO3.CMS/+/93502 -
Christian Kuhn wrote 06/18/26, 23:10
Removed reviewer Christian Kuhn with the following votes: * Verified+1 by Christian Kuhn * Code-Review+1 by Christian Kuhn -
Simon Schaufelberger voted on patch set 5 CR +1 V +1 06/26/26, 12:54
-
Garvin Hicking voted on patch set 5 CR +2 V +2 06/26/26, 16:35
-
Garvin Hicking merged patch set 6 07/15/26, 14:41