Review main Merged

93498[TASK] Speed up responsive image loading from db

Ayke Halder CR +2 V +2 Hard +176 −99 updated 07/15/26, 14:41

Commit message

Rendering responsive images creates many variants of the
same original, for instance the widths of a `srcset`.
So far each variant lookup issued its own query to
`sys_file_processedfile`, so a page with a handful of
images in several variants quickly added up to hundreds
of queries.

We now load all variants of an original at once and keep
the rows in the runtime cache. Looking up further variants
of the same already processed original is then served from
that cache instead of the database. This is independent of
how the images are rendered and works for any combination
of controller logic, Fluid ViewHelpers, Fluid loops and
TypoScript `sourceCollection` configuration.

The runtime cache is kept coherent through tagged entries.
Adding, updating and removing a processed file flush
entries of the affected original, and `removeAll()` drops
the whole cache in one go.
The direct table delete in FileDeletionAspect is replaced
by a new remove() method so this path is covered too.
CleanUpLocalProcessedFilesService is moved to constructor
dependency injection and tidied up along the way and
moves a method into the repository, too.

When no responsive images are used and only a single
variant exists, the added overhead is negligible.

Review

  • Code review CR +2 Done
    • Garvin Hicking +2
    • Simon Schaufelberger +1
  • Verified V +2 Done
    • Garvin Hicking +2
    • Simon Schaufelberger +1
    • core-ci +1 CI

Issues 1

Files 5

Review log 2328

Show 23 earlier messages
  1. Ayke Halder uploaded patch set 1 03/31/26, 16:59

  2. core-ci voted on patch set 1 V +1 03/31/26, 17:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99298

  3. Sven Liebert commented on patch set 1 CR +1 V +1 04/02/26, 09:24

    I think we can backport that to v13 according to what @benni@typo3.org wrote in slack
  1. Ayke Halder commented on patch set 1 04/02/26, 09:30

    Reply

    Commit message changed: * v13 * make clear that this speeds up loading from db - and not browsers loading the images.
  2. Ayke Halder updated the commit message in patch set 2 CR +1 V +1 04/02/26, 09:32

  3. core-ci voted on patch set 2 V +1 04/02/26, 09:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/99462

  4. Sven Liebert voted on patch set 2 CR +1 V +1 04/02/26, 09:54

  5. Benni Mack updated the commit message in patch set 3 CR +1 V +1 04/22/26, 06:44

  6. core-ci voted on patch set 3 V +1 04/22/26, 11:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/100782

  7. Christian Kuhn commented on patch set 3 06/11/26, 23:35

    thanks for this work! patch is good in general: this is one of the rarer places where db results can be runtime-cached effectively and still be kept coherent, *if* we centralize all db operations into the repository. i adapted two further cases and cleaned up everything a bit.
  8. Christian Kuhn uploaded patch set 4 V +1 06/11/26, 23:38

  9. core-ci voted on patch set 4 V +1 06/11/26, 23:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/103414

  10. Christian Kuhn voted on patch set 4 CR +1 V +1 06/11/26, 23:49

  11. Ayke Halder commented on patch set 4 06/12/26, 05:58

    CleanUpLocalProcessedFilesService.php:206

    Is the logic change intentional? E.g. should the edge case `configuration['pathType'] = 0` be considered? To not change the current logic use elvis operator `?:` here.
  12. Ayke Halder commented on patch set 4 06/12/26, 06:01

    CleanUpLocalProcessedFilesService.php:206 Reply

    … `configuration['basePath'] = 0` …
  13. Garvin Hicking commented on patch set 4 V +1 06/12/26, 07:40

    This works for me. Queries down from 11 to 1, same output. Seems solid, just one question and the issue Ayke raised and then I'm good to vote/merge 😊

    ProcessedFileRepository.php:186

    Maybe for main we can think about dropping this serialization; we could use json_encode here as it's a lot faster (3msec vs. 30msec on 100_000 iterations), and I would like to prevent utilizations of serialize() where unserialization is not needed. I recognize that the stored sha1 then would change and invalidate all create processed files, so that's a kind of breaking change. Thoughts (follow-up patch, of course)?

    CleanUpLocalProcessedFilesService.php:206 Reply

    I agree, we should use ?: here
  14. Ayke Halder commented on patch set 4 V -1 06/12/26, 09:51

    One point that gives me a headache: Let's say we have 100.000 Users and we create some badge-grapic with the username on top. As all these 100.000 badges are based on the same image file, and only differ in image processing instructions, with this change all the 100.000 variations get loaded from db - even when only a single image is needed. Please proof me wrong on this - or declare it a super-rare edge case.
  15. Christian Kuhn commented on patch set 4 06/12/26, 10:22

    ProcessedFileRepository.php:186 Reply

    yes, we need to think about it, there are issues on it, too. separate change, though.
  16. Christian Kuhn commented on patch set 4 06/12/26, 10:25

    Reply

    imho super edgy, and probably still not *that* slow when it's one query. goal is fe performance with the patch, and that's what we should look at primarily. if this *really* turns out to be an issue, we'll reconsider to find a solution.
  17. Christian Kuhn uploaded patch set 5 CR +1 V +1 06/12/26, 11:20

  18. Christian Kuhn commented on patch set 5 CR +1 V +1 06/12/26, 11:21

    CleanUpLocalProcessedFilesService.php:206 Reply

    Done
  19. core-ci voted on patch set 5 V +1 06/12/26, 11:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/103444

  20. Garvin Hicking commented on patch set 5 CR +1 V +1 06/12/26, 12:38

    Reply

    I made a test and created 200k variants of one original file. With 14 rows it took 0.16 microseconds to query and populate cache. With these 200k rows it took 294 microseconds. That is a huge relative increase I agree, however in the broad terms of page assembly for this huge amount of rows, 0.2 seconds is not really that bad of an impact, especially when at least for frontend rendering that result is cached, and for extension / badge usage scenarios, creating such an overlay might also be subject to caching. I think with relevant scenarios, this is a larger win than hit for this edge-scenario, but I'm happy you brought this up and thought about this, I found that very interesting to check out.
  21. Ayke Halder commented on patch set 5 06/13/26, 15:37

    Reply

    @lolli@schwarzbu.ch @garvin@hick.ing Thanks for looking into this. So I also think it's safe to merge this now.

    ProcessedFileRepository.php:186 Reply

    @garvin@hick.ing Have a look at: 93502: [TASK] Migrate processed file configuration storage from serialize() to JSON | https://review.typo3.org/c/Packages/TYPO3.CMS/+/93502
  22. Christian Kuhn wrote 06/18/26, 23:10

    Removed reviewer Christian Kuhn with the following votes: * Verified+1 by Christian Kuhn * Code-Review+1 by Christian Kuhn
  23. Simon Schaufelberger voted on patch set 5 CR +1 V +1 06/26/26, 12:54

  24. Garvin Hicking voted on patch set 5 CR +2 V +2 06/26/26, 16:35

  25. Garvin Hicking merged patch set 6 07/15/26, 14:41