Merge conflict Gerrit cannot merge this change. It needs a rebase onto main.

Review main Work in progress Quiet

62956[WIP][TASK] Improves redirect performance with large redirects table

Mario Lubenka CR -1 V -1 Expert +568 −634 updated 10/08/26, 17:29

Commit message

Redirects has been cached as one prepared array structure. Thus this
array has been grown quite large on instances with a high number of
redirects. Furthermore with growing count of redirects, preparing
and rewriting the cache array slows down operations in the backend
and memory needs are increased, not only throug retrieving and
writing in backend, but also in the frontend on each request.

Preparing and caching redirects has been implemented believing it
would mitigate performance issues and to create a way of process
redirects in a defined prioritized way.

Choosen preperation has been looked as a good way to do so, which
is misleading. The biggest caveat is that all instance redirects
has to been loaded into memory, which depends on avaiable memory
and massivly depends on block io and cpu power.

Thinking about instances, that means that in that all domains
plus all domains (*) redirects has been placed in that cache.

Omitting to describe cache structure here, but has been choosen
to allow a though performanc matching process in the redirect
frontend middleware.

Matching has been done in following schema:

 1. source_host === <current_host>
    1. find possibleRedirects
        1. add if path + query match (respect_query_parameters)
        2. addi path patch an entry (flat)
        3. iterate all regexp and add all that matches
    2. loop possibleRedirects
        - check if active (not disabled and evaluate start/end )
          -> return first active item

 2. source_host === *
    1. find possibleRedirects
        1. path + query match an entry
        2. path patch an entry
        3. iterate all regexp, validation them
    2. loop possibleRedirects
        - check if active (not disabled and evaluate start/end )
          -> return first active item

To mitigiate the memory and processing issue as far as possible
an efficte query is created to maintain the matching logic but
filtering out all unnessecary redirects direct on the database.
With the implemented query high numbers of redirects in a
instance should be no issue anymore, except RegExp redirects.

RegExp redirects cannot be pre-filtered and has to be evaluated
after retrieving them. Instead of retrieving the full result
set into the memory, curser based retrieving is used.

High number of RegExp redirect processing still takes time, even
if no match is found.

RedirectCache and Service, also DataHandler hook for cache rebuild
is removed and replaced by a database query.

------------

ToDo's:

[x] Instead of retrieve all results at once, use cursor
[~] Ensure correct condition for query: need test coverage
[~] Ensure correct ordering for query: need tests coverage
[x] Ensure deterministic query
[ ] Add more functional tests to cover constellations and
    priority sorting
[ ] Add more functional tests to test matching deeply
[ ] Create meaningfull comparision data and benchmark before/after
    to have a validation of the improvement; this should not only
    cover the 'a lot of redirects' case
[ ] Change commit message to describe what is done, and why with
    providing the background and how it is archieved. And further
    add needed background information to understand how redirects
    works or should work.
[ ] Add breaking REsT file describing the change and why
    RedirectCacheService and RedirectCache is dropped.
[ ] Test against all dbms and versions runTests.sh provides

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 17
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/56/62956/17 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/56/62956/17 && git cherry-pick FETCH_HEAD

Review

  • Code review CR -1 Needs a vote
    • Stefan Bürk -1
  • Verified V -1 Blocked
    • core-ci -1 CI

Issues 1

Files 12

Review log 4563

Show 2538 earlier messages
  1. Mario Lubenka uploaded patch set 1 01/17/20, 15:49

  2. TYPO3com voted on patch set 1 V -1 01/17/20, 15:49 Completed build in 14s on Fri, 17 Jan, 03:49 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40065

  3. Mario Lubenka rebased onto patch set 2 01/17/20, 15:51

  4. TYPO3com voted on patch set 2 V -1 01/17/20, 15:52 Completed build in 14s on Fri, 17 Jan, 03:52 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40066

  5. Mario Lubenka uploaded patch set 3 01/17/20, 15:53

  6. TYPO3com voted on patch set 3 V -1 01/17/20, 15:54 Completed build in 14s on Fri, 17 Jan, 03:54 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40067

  7. Mario Lubenka commented on patch set 3 01/17/20, 16:04

    I didn't find a Documentation directory in the extension. Where do I put the Changelog file?
  8. Markus Klein commented on patch set 3 01/17/20, 16:20

    can you please post numbers how the performance is improved with this implementation? And also the necessary indexes for the DB table, if any.
  9. Mario Lubenka commented on patch set 3 01/20/20, 14:38

    Reply

    I added some performance numbers in the ticket.
  10. Daniel Goerz commented on patch set 3 CR -1 02/15/20, 11:27

    RedirectService.php:116

    Just keep the default restrictions for this approach

    RedirectService.php:182

    This last block of constraints is not needed at all, just remove the fiddling with the Restrictions above, the default restrictions do exactly what you are doing here as well
  11. TYPO3com commented on patch set 3 02/15/20, 11:34 Completed build in 23s on Sat, 15 Feb, 11:34 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-109

  12. Daniel Goerz uploaded patch set 4 02/15/20, 13:09

  13. TYPO3com voted on patch set 4 V -1 02/15/20, 13:10 Completed build in 21s on Sat, 15 Feb, 01:10 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-115

  14. Daniel Goerz commented on patch set 4 CR -1 02/15/20, 13:17

    In general it is correct, that with many many redirects the current approach of fetching all redirects from the cache becomes very slow compared to the DB-Query suggested in the patch. However, in the current state we lose the priorization of redirect candidates. The removed unit tests no longer make sense because of this. We should reimplement the order in which redirects are respected and restore the removed tests as unit or functional tests again.
  15. Daniel Goerz uploaded patch set 5 02/15/20, 13:18

  16. Daniel Goerz commented on patch set 5 CR -1 02/15/20, 13:18

    Reapply my -1
  17. TYPO3com voted on patch set 5 V -1 02/15/20, 13:31 Completed build in 13m 22s on Sat, 15 Feb, 01:31 PM Test Summary: 30 of 71703 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-116

  18. Daniel Goerz uploaded patch set 6 02/15/20, 13:37

  19. TYPO3com voted on patch set 6 V +1 02/15/20, 13:51 Completed build in 13m 42s on Sat, 15 Feb, 01:51 PM Test Summary: 71245 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-117

  20. Daniel Goerz voted on patch set 6 CR -1 02/15/20, 13:54

  21. Benni Mack commented on patch set 6 04/06/20, 09:54

    Could somebody please re-base this patch?
  22. Markus Klein uploaded patch set 7 04/06/20, 22:38

  23. Markus Klein uploaded patch set 8 04/06/20, 22:43

  24. Markus Klein commented on patch set 8 04/06/20, 22:43

    RedirectRepository.php:222

    I think loading all redirects into memory might not be a good idea. Better return the cursor to the caller
  25. TYPO3com voted on patch set 7 V +1 04/06/20, 22:50 Completed build in 11m 58s on Mon, 6 Apr, 10:50 PM Test Summary: 73315 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2243

  26. TYPO3com voted on patch set 8 V +1 04/06/20, 22:56 Completed build in 12m 45s on Mon, 6 Apr, 10:56 PM Test Summary: 73315 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2244

  27. Stefan Bürk uploaded patch set 9 09/07/21, 10:29

  28. Stefan Bürk commented on patch set 9 CR -1 09/07/21, 10:31

    Only rebased, will look deeper into it
  29. core-ci voted on patch set 9 V +1 09/07/21, 10:46 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16370

  30. Stefan Bürk uploaded patch set 10 09/11/21, 12:57

  31. Stefan Bürk commented on patch set 10 CR -1 09/11/21, 12:58

    Still on it, just a between push & adding todos to commit message for now
  32. Stefan Bürk uploaded patch set 11 09/11/21, 13:16

  33. Stefan Bürk commented on patch set 11 CR -1 09/11/21, 13:16

    solving merge conflict
  34. core-ci voted on patch set 10 V +1 09/11/21, 13:20 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16556

  35. core-ci voted on patch set 11 V +1 09/11/21, 13:32 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16558

  36. Stefan Bürk uploaded patch set 12 09/11/21, 21:50

  37. core-ci voted on patch set 12 V -1 09/11/21, 22:07 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16582

  38. Stefan Bürk uploaded patch set 13 09/11/21, 23:55

  1. core-ci voted on patch set 13 V +1 09/12/21, 00:14 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16583

  2. Stefan Bürk rebased onto patch set 14 09/12/21, 12:16

  3. core-ci voted on patch set 14 V -1 09/12/21, 12:25 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16617

  4. Stefan Bürk voted on patch set 14 CR -1 09/12/21, 20:14

  5. Stefan Bürk rebased onto patch set 15 09/20/21, 21:46

  6. core-ci voted on patch set 15 V +1 09/20/21, 22:00 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16957

  7. Benni Mack wrote 11/29/21, 10:08

    Change destination moved from master to main
  8. Benni Mack updated the commit message in patch set 16 11/29/21, 13:28

  9. core-ci voted on patch set 16 V +1 11/29/21, 14:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20034

  10. Mario Lubenka uploaded patch set 17 04/21/22, 07:24

  11. Mario Lubenka commented on patch set 17 04/21/22, 07:26

    Rebased onto main. Wasn't able to install the testing-framework right now so I didn't run the tests before pushing.

    RedirectRepository.php:222 Reply

    Done

    RedirectService.php:116 Reply

    Done

    RedirectService.php:182 Reply

    Done
  12. core-ci voted on patch set 17 V -1 04/21/22, 07:33 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26264

  13. Stefan Bürk commented on patch set 17 04/21/22, 08:39

    rebase conflict solving not finished

    Reply

    That's fine, will have a look later. Had this patch on my todo list to look further into it (making some benchmarkings) and event further improve it if needed. Are you eager to work further on that patch ? Maybe I can share some thoughts on that with you in a slack call - otherwise I would pick it up. However, thanks for rebasing it - left that out intended until I work again on that patch. Rebase went kind of wrong or at least was not finished properly - granted that there has been a lot of changes to deal with in a rebase conflict.

    RedirectRepository.php:192 Unresolved

    Would suggest to use $queryBuilder->expr()->or() here. Same to places below, and `$queryBuillder->expr()->and()` for the and variant. It's more to keep the things the way it has been done - and keep people firm using ExpressionBuilder first. However using static CompositeMethods directly are technically correct.
  14. Mario Lubenka commented on patch set 17 04/21/22, 09:30

    Reply

    Oh right. I didn't see much movement here anymore, so I felt obliged to do something about this as I initiated the PR. ^^ I'd welcome you to working on this instead.
  15. Stefan Bürk commented on patch set 17 04/21/22, 09:35

    Reply

    Yeah. However, it was clear that this will be a v12 change, so delayed it a bit and worked quite some other redirect changes first. I have this patch on my todo list. But it needs further tests (performance in different constellations etc etc). Because regexp redirects are not handled on the server side, thus eventually needs to retrieve a large(r) result set .. which may be become quit fast a slowdown in contrast to the cache file. Don't get me wrong - the idea behind is okay, but this needs to be done quite by hearth and with a lot of statistical measurement - not only code changes and simple "works" evaluation. If it is fine for you I would continue on this - but cannot not say when exactly. But goal is to have it, if possible, in v12.
  16. Stefan Bürk voted on patch set 17 CR -1 04/21/22, 09:36

  17. Oliver Klee commented on patch set 17 04/21/22, 10:01

    Unresolved

    Commit message nitpick for the next version of this changeset: Improves -> Improve
  18. Mario Lubenka commented on patch set 17 02/08/23, 12:53

    Reply

    With the upcoming feature freeze for v12, is this change still considered to be part of the LTS? 😊
  19. Stefan Bürk commented on patch set 17 02/08/23, 13:39

    Reply

    I don't think it will make into 12LTS. We have done some other stuff in redirects or at least are currently in the pipeline. Some minro will follow. Otherwise my time is full with something other which is considered essentially for v12 (doctrine/dbal 4) - or at least to be ready. The main issue is, to switch to the database and removing the cache, it must be possible that regexp-redirects are correctly matched and handled by the database directly. only retrieving them would make it drastically unreliable for people with quite some regexp redirects out there .. and doing both does not make much sense. But there have been other intermediate steps - for example, there are now (quit a while) dedicated cached per source host (one for "*", one for "domain1.tld" and so on) instead of one huge one. Sure .. one domain with a lot of redirects will still need time to load the file. However, there are events in the pipeline one to implement a custom matching - we could consider to add a additional query where you can ommit the cache file writing additionally. (redirects are loaded before the event right now) - maybe rethinking the event to switch them (loading redirects only if event do not return a matched redirect): https://review.typo3.org/c/Packages/TYPO3.CMS/+/77675 But I have that still on the list - there are must have requirements which must work. Performance check with different load situation must be done then also. Guess this will be more thing for v13, if possible. I mean - if you want you can work on it and we will see .. put personally I do not have the time to finish this. Sorry for that.
  20. Mario Lubenka commented on patch set 17 02/08/23, 13:45

    Reply Unresolved

    No worries. We're mainly working with HAproxy redirects which does not involve TYPO3 anyways. So we're not needing it for TYPO3 v12.
  21. Oliver Klee wrote 08/25/24, 10:14

    Removed cc Oliver Klee.
  22. Benni Mack wrote 03/22/25, 16:41

    Topic set to redirects
  23. Benni Mack wrote 04/21/26, 14:56

    Removed cc Benni Mack.
  24. Benni Mack marked it work in progress 04/21/26, 14:56

  25. Benni Mack commented on patch set 17 10/08/26, 17:29

    Unresolved

    lots of things have changed, I made some improvements here: https://review.typo3.org/c/Packages/TYPO3.CMS/+/96538 and would recommend that one over this approach which seems kinda stuck since 6years, I really feel sorry about that.