Review main Ready for review

96538[TASK] Improve redirect matching performance for many redirects

Benni Mack CR +1 V +1 Expert +750 −490 updated 11/20/26, 20:02

Commit message

Every frontend request checked for a redirect loaded all redirects
of the requested host and of the wildcard host from a single cache
entry, built from the whole sys_redirect table. With many redirects
this entry became slow to load, rebuild and unserialize and could
exceed the available memory: 400k redirects took about one second
and more than 1 GB per request.

Redirects without a regular expression are now looked up with an
indexed query on source_host and source_path. The cache entry of a
host only keeps the regular expression redirects, the rare plain
redirects with multiple trailing slashes, and a Bloom filter of all
other source paths. The filter rules out a match for about 99 % of
the requests not matching a redirect, so regular page views still
do not query sys_redirect at all.

The matching order is unchanged and now covered by functional tests
running against a real cache backend. With 400k redirects a lookup
takes 1 to 4 ms and about 100 MB instead.

Benchmark (400k redirects, database cache backend; old -> new):

| case          | SQLite         | MariaDB        | Postgres       |
|---------------|----------------|----------------|----------------|
| no match      | 1011 -> 1.2 ms | 1170 -> 1.4 ms | 1036 -> 2.7 ms |
| plain match   | 836 -> 2.3 ms  | 823 -> 3.0 ms  | 786 -> 3.5 ms  |
| cache rebuild | 1164 -> 263 ms | 1382 -> 387 ms | 1096 -> 268 ms |
| peak memory   | 1152 -> 104 MB | 1136 -> 110 MB | 1152 -> 105 MB |

At 1k redirects a miss now takes 0.15-0.3 ms instead of 2.2-3.4 ms.

Rebuilding the cache of an empty source host now updates the
wildcard entry, which is the one read when matching. Before, it
wrote an entry which was never read, leaving the wildcard redirects
outdated.

Tradeoffs:
- Rebuilds still read every source path of a host, because the
  filter needs them. The DataHandler hook rebuilds on every redirect
  save, so saving one redirect costs about 0.3 s at 400k (old code:
  over 1 s). If that matters, the hook could just flush the entry
  and let the next frontend request rebuild it.
- New source paths of plain redirects only become matchable after
  the cache rebuild, as before. Other changes to plain redirects,
  e.g. their target or visibility, now apply immediately.

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 5
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/38/96538/5 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/38/96538/5 && git cherry-pick FETCH_HEAD

Review

  • Code review CR +1 Needs a vote
    • Georg Ringer +1
  • Verified V +1 Needs a vote
    • Georg Ringer +1
    • core-ci +1 CI

Merge conflicts 4

Issues 1

Files 8

Review log 1116

  1. Benni Mack uploaded patch set 1 10/08/26, 17:22

  2. Benni Mack updated the commit message in patch set 2 10/08/26, 17:28

  3. core-ci voted on patch set 1 V +1 10/08/26, 17:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110520

  4. core-ci voted on patch set 2 V +1 10/08/26, 17:36 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110521

  5. Benni Mack uploaded patch set 3 V +1 10/16/26, 09:48

  6. Benni Mack marked it ready for review 10/16/26, 09:49

  7. core-ci voted on patch set 3 V -1 10/16/26, 10:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111345

  8. Benni Mack uploaded patch set 4 V -1 10/16/26, 14:45

  9. core-ci voted on patch set 4 V +1 10/16/26, 14:53 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/111369

  10. Markus Klein commented on patch set 4 10/20/26, 15:34

    Nice one. Still need to review a bit more

    RedirectService.php:308

    ParameterType::INTEGER
  11. Daniel Windloff commented on patch set 4 10/25/26, 09:01

    Very nice improvement.

    BloomFilter.php:101

    I think `xxh3` could be better here. It's faster, and if I compare the false positive rate for a bloom filter with 10^5 entries and 10^7 not matching test cases, the false positive rate seems to be a little bit lower on average. I'm not totally sure, as I never implemented a bloom filter in php, but I assume `unpack` `V2` matches better with `xxh3` and is not so good with `xxh128`.
  12. Benni Mack commented on patch set 4 10/30/26, 21:35

    RedirectService.php:308 Reply

    Let's change this separately please, if we want this. Current core: 17 x ParameterType::INTEGER 857 x Connection::PARAM_INT
  13. Benni Mack uploaded patch set 5 V +1 10/30/26, 21:54

  14. Benni Mack commented on patch set 5 10/30/26, 21:59

    BloomFilter.php:101 Reply

    Thanks! I did a quick test: 100k keys, 2M non-matching lookups, 3 runs each. False positive rates were 1.00–1.01 % for both xxh128 and xxh3, at the same speed (~0.47 µs per lookup); This was just done locally on my laptop. But no measurable difference, but since unpack('V2') only uses 64 bits anyway, your suggestion is better.
  15. core-ci voted on patch set 5 V +1 10/30/26, 22:03 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112919

  16. Georg Ringer voted on patch set 5 CR +1 V +1 11/20/26, 20:02