Merge conflict Gerrit cannot merge this change. It needs a rebase onto main.
62956[WIP][TASK] Improves redirect performance with large redirects table
Commit message
Test this change
Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch
- Check out patch set 17
-
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/56/62956/17 && git checkout FETCH_HEAD - Cherry-pick onto main
-
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/56/62956/17 && git cherry-pick FETCH_HEAD
Review
-
Code review CR -1 Needs a vote
- Stefan Bürk -1
-
Verified V -1 Blocked
- core-ci -1 CI
Issues 1
Files 12
- typo3/sysext/redirects/Classes/Hooks/DataHandlerCacheFlushingHook.phpdeleted+0 −92
- typo3/sysext/redirects/Classes/Repository/RedirectRepository.php3+106 −1
- typo3/sysext/redirects/Classes/Service/IntegrityService.php+3 −2
- typo3/sysext/redirects/Classes/Service/RedirectCacheService.phpdeleted+0 −122
- typo3/sysext/redirects/Classes/Service/RedirectService.php4+36 −91
- typo3/sysext/redirects/Tests/Functional/Repository/Fixtures/RedirectsRepositoryTest_redirectscomplexsorting.xmlnew+195 −0
- typo3/sysext/redirects/Tests/Functional/Repository/Fixtures/RediretsRepositoryTest_redirectshostsorting.xmlnew+29 −0
- typo3/sysext/redirects/Tests/Functional/Repository/RedirectRepositoryTest.php+165 −0
- typo3/sysext/redirects/Tests/Functional/Service/IntegrityServiceTest.php+0 −2
- typo3/sysext/redirects/Tests/Functional/Service/RedirectServiceTest.php+3 −3
- typo3/sysext/redirects/Tests/Unit/Service/RedirectServiceTest.php+29 −317
- typo3/sysext/redirects/ext_localconf.php+2 −4
Review log 4563
Show 2538 earlier messages
-
Mario Lubenka uploaded patch set 1 01/17/20, 15:49
-
TYPO3com voted on patch set 1 V -1 01/17/20, 15:49 Completed build in 14s on Fri, 17 Jan, 03:49 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40065
-
Mario Lubenka rebased onto patch set 2 01/17/20, 15:51
-
TYPO3com voted on patch set 2 V -1 01/17/20, 15:52 Completed build in 14s on Fri, 17 Jan, 03:52 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40066
-
Mario Lubenka uploaded patch set 3 01/17/20, 15:53
-
TYPO3com voted on patch set 3 V -1 01/17/20, 15:54 Completed build in 14s on Fri, 17 Jan, 03:54 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-40067
-
Mario Lubenka commented on patch set 3 01/17/20, 16:04
I didn't find a Documentation directory in the extension. Where do I put the Changelog file? -
Markus Klein commented on patch set 3 01/17/20, 16:20
can you please post numbers how the performance is improved with this implementation? And also the necessary indexes for the DB table, if any. -
Mario Lubenka commented on patch set 3 01/20/20, 14:38
Reply
I added some performance numbers in the ticket. -
Daniel Goerz commented on patch set 3 CR -1 02/15/20, 11:27
RedirectService.php:116Just keep the default restrictions for this approachRedirectService.php:182This last block of constraints is not needed at all, just remove the fiddling with the Restrictions above, the default restrictions do exactly what you are doing here as well -
TYPO3com commented on patch set 3 02/15/20, 11:34 Completed build in 23s on Sat, 15 Feb, 11:34 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-109
-
Daniel Goerz uploaded patch set 4 02/15/20, 13:09
-
TYPO3com voted on patch set 4 V -1 02/15/20, 13:10 Completed build in 21s on Sat, 15 Feb, 01:10 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-115
-
Daniel Goerz commented on patch set 4 CR -1 02/15/20, 13:17
In general it is correct, that with many many redirects the current approach of fetching all redirects from the cache becomes very slow compared to the DB-Query suggested in the patch. However, in the current state we lose the priorization of redirect candidates. The removed unit tests no longer make sense because of this. We should reimplement the order in which redirects are respected and restore the removed tests as unit or functional tests again. -
Daniel Goerz uploaded patch set 5 02/15/20, 13:18
-
Daniel Goerz commented on patch set 5 CR -1 02/15/20, 13:18
Reapply my -1 -
TYPO3com voted on patch set 5 V -1 02/15/20, 13:31 Completed build in 13m 22s on Sat, 15 Feb, 01:31 PM Test Summary: 30 of 71703 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-116
-
Daniel Goerz uploaded patch set 6 02/15/20, 13:37
-
TYPO3com voted on patch set 6 V +1 02/15/20, 13:51 Completed build in 13m 42s on Sat, 15 Feb, 01:51 PM Test Summary: 71245 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-117
-
Daniel Goerz voted on patch set 6 CR -1 02/15/20, 13:54
-
Benni Mack commented on patch set 6 04/06/20, 09:54
Could somebody please re-base this patch? -
Markus Klein uploaded patch set 7 04/06/20, 22:38
-
Markus Klein uploaded patch set 8 04/06/20, 22:43
-
Markus Klein commented on patch set 8 04/06/20, 22:43
RedirectRepository.php:222I think loading all redirects into memory might not be a good idea. Better return the cursor to the caller -
TYPO3com voted on patch set 7 V +1 04/06/20, 22:50 Completed build in 11m 58s on Mon, 6 Apr, 10:50 PM Test Summary: 73315 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2243
-
TYPO3com voted on patch set 8 V +1 04/06/20, 22:56 Completed build in 12m 45s on Mon, 6 Apr, 10:56 PM Test Summary: 73315 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-2244
-
Stefan Bürk uploaded patch set 9 09/07/21, 10:29
-
Stefan Bürk commented on patch set 9 CR -1 09/07/21, 10:31
Only rebased, will look deeper into it -
core-ci voted on patch set 9 V +1 09/07/21, 10:46 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16370
-
Stefan Bürk uploaded patch set 10 09/11/21, 12:57
-
Stefan Bürk commented on patch set 10 CR -1 09/11/21, 12:58
Still on it, just a between push & adding todos to commit message for now -
Stefan Bürk uploaded patch set 11 09/11/21, 13:16
-
Stefan Bürk commented on patch set 11 CR -1 09/11/21, 13:16
solving merge conflict -
core-ci voted on patch set 10 V +1 09/11/21, 13:20 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16556
-
core-ci voted on patch set 11 V +1 09/11/21, 13:32 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16558
-
Stefan Bürk uploaded patch set 12 09/11/21, 21:50
-
core-ci voted on patch set 12 V -1 09/11/21, 22:07 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16582
-
Stefan Bürk uploaded patch set 13 09/11/21, 23:55
-
core-ci voted on patch set 13 V +1 09/12/21, 00:14 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16583
-
Stefan Bürk rebased onto patch set 14 09/12/21, 12:16
-
core-ci voted on patch set 14 V -1 09/12/21, 12:25 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16617
-
Stefan Bürk voted on patch set 14 CR -1 09/12/21, 20:14
-
Stefan Bürk rebased onto patch set 15 09/20/21, 21:46
-
core-ci voted on patch set 15 V +1 09/20/21, 22:00 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16957
-
Benni Mack wrote 11/29/21, 10:08
Change destination moved from master to main -
Benni Mack updated the commit message in patch set 16 11/29/21, 13:28
-
core-ci voted on patch set 16 V +1 11/29/21, 14:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20034
-
Mario Lubenka uploaded patch set 17 04/21/22, 07:24
-
Mario Lubenka commented on patch set 17 04/21/22, 07:26
Rebased onto main. Wasn't able to install the testing-framework right now so I didn't run the tests before pushing.RedirectRepository.php:222ReplyDoneRedirectService.php:116ReplyDoneRedirectService.php:182ReplyDone -
core-ci voted on patch set 17 V -1 04/21/22, 07:33 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26264
-
Stefan Bürk commented on patch set 17 04/21/22, 08:39
rebase conflict solving not finishedReply
That's fine, will have a look later. Had this patch on my todo list to look further into it (making some benchmarkings) and event further improve it if needed. Are you eager to work further on that patch ? Maybe I can share some thoughts on that with you in a slack call - otherwise I would pick it up. However, thanks for rebasing it - left that out intended until I work again on that patch. Rebase went kind of wrong or at least was not finished properly - granted that there has been a lot of changes to deal with in a rebase conflict.RedirectRepository.php:192UnresolvedWould suggest to use $queryBuilder->expr()->or() here. Same to places below, and `$queryBuillder->expr()->and()` for the and variant. It's more to keep the things the way it has been done - and keep people firm using ExpressionBuilder first. However using static CompositeMethods directly are technically correct. -
Mario Lubenka commented on patch set 17 04/21/22, 09:30
Reply
Oh right. I didn't see much movement here anymore, so I felt obliged to do something about this as I initiated the PR. ^^ I'd welcome you to working on this instead. -
Stefan Bürk commented on patch set 17 04/21/22, 09:35
Reply
Yeah. However, it was clear that this will be a v12 change, so delayed it a bit and worked quite some other redirect changes first. I have this patch on my todo list. But it needs further tests (performance in different constellations etc etc). Because regexp redirects are not handled on the server side, thus eventually needs to retrieve a large(r) result set .. which may be become quit fast a slowdown in contrast to the cache file. Don't get me wrong - the idea behind is okay, but this needs to be done quite by hearth and with a lot of statistical measurement - not only code changes and simple "works" evaluation. If it is fine for you I would continue on this - but cannot not say when exactly. But goal is to have it, if possible, in v12. -
Stefan Bürk voted on patch set 17 CR -1 04/21/22, 09:36
-
Oliver Klee commented on patch set 17 04/21/22, 10:01
Unresolved
Commit message nitpick for the next version of this changeset: Improves -> Improve -
Mario Lubenka commented on patch set 17 02/08/23, 12:53
Reply
With the upcoming feature freeze for v12, is this change still considered to be part of the LTS? 😊 -
Stefan Bürk commented on patch set 17 02/08/23, 13:39
Reply
I don't think it will make into 12LTS. We have done some other stuff in redirects or at least are currently in the pipeline. Some minro will follow. Otherwise my time is full with something other which is considered essentially for v12 (doctrine/dbal 4) - or at least to be ready. The main issue is, to switch to the database and removing the cache, it must be possible that regexp-redirects are correctly matched and handled by the database directly. only retrieving them would make it drastically unreliable for people with quite some regexp redirects out there .. and doing both does not make much sense. But there have been other intermediate steps - for example, there are now (quit a while) dedicated cached per source host (one for "*", one for "domain1.tld" and so on) instead of one huge one. Sure .. one domain with a lot of redirects will still need time to load the file. However, there are events in the pipeline one to implement a custom matching - we could consider to add a additional query where you can ommit the cache file writing additionally. (redirects are loaded before the event right now) - maybe rethinking the event to switch them (loading redirects only if event do not return a matched redirect): https://review.typo3.org/c/Packages/TYPO3.CMS/+/77675 But I have that still on the list - there are must have requirements which must work. Performance check with different load situation must be done then also. Guess this will be more thing for v13, if possible. I mean - if you want you can work on it and we will see .. put personally I do not have the time to finish this. Sorry for that. -
Mario Lubenka commented on patch set 17 02/08/23, 13:45
Reply Unresolved
No worries. We're mainly working with HAproxy redirects which does not involve TYPO3 anyways. So we're not needing it for TYPO3 v12. -
Oliver Klee wrote 08/25/24, 10:14
Removed cc Oliver Klee. -
Benni Mack wrote 03/22/25, 16:41
Topic set to redirects -
Benni Mack wrote 04/21/26, 14:56
Removed cc Benni Mack. -
Benni Mack marked it work in progress 04/21/26, 14:56
-
Benni Mack commented on patch set 17 10/08/26, 17:29
Unresolved
lots of things have changed, I made some improvements here: https://review.typo3.org/c/Packages/TYPO3.CMS/+/96538 and would recommend that one over this approach which seems kinda stuck since 6years, I really feel sorry about that.