Review main Merged

73118[BUGFIX] Use clustered cache entries per domain for redirects

Stefan Bürk CR +2 V +2 Hard +201 −104 updated 05/09/22, 11:09

Commit message

'ext:redirect' now clusters the redirect cache into dedicated
parts per domain, which reduces the data amount needed to load
on redirect handling in the frontend to the absolute minimum.

On the other hand, when rebuilding a cache this is narrowed down
to the related domain, thus avoiding retrieving and caching all
redirects, which may reduce the processing time in the backend
on changing redirects or auto redirects creation on page updates.

This works around some flaws for some instances and use cases,
but do not harm on smaller instances with lower redirects count.

* read and write redirect cache by domain based identifier
* removed one array level from cached information and reduced
  one array key level in redirect matching code
* cache identifier for domain entries are build using 'sha1()'
* combined with early redirects introduced with #96480 this
  reduces the loaded data even more, as it is done in chunks
* adjusted unit tests to reflect two redirect cache service
  calls, one for the domain and one for wildcard "*".
* rebuilding cache through datahandler hook now rebuilds
  only for the related 'source_host', thus avoiding rebuild
  of all redirects for every change. This may still be further
  improved for multi record operations with the same domain.
  A 'rebuildAll()' fallback stayed as a last defense to cover
  cases where specific source_host cannot be determined.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Oli Bartsch +1
    • Stefan Bürk +1
  • Verified V +2 Done
    • Benni Mack +2
    • Oli Bartsch +1
    • core-ci +1 CI

Issues 1

Files 6

Review log 3451

Show 1726 earlier messages
  1. Stefan Bürk uploaded patch set 1 01/21/22, 17:05

  2. core-ci voted on patch set 1 V -1 01/21/22, 17:07 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22363

  3. Stefan Bürk uploaded patch set 2 01/21/22, 21:19

  4. core-ci voted on patch set 2 V +1 01/21/22, 21:32 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22378

  5. Stefan Bürk uploaded patch set 3 01/21/22, 23:49

  6. core-ci voted on patch set 3 V +1 01/22/22, 00:01 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22390

  7. Stefan Bürk uploaded patch set 4 01/24/22, 00:48

  8. Stefan Bürk marked it work in progress 01/24/22, 00:49

  9. core-ci voted on patch set 4 V +1 01/24/22, 01:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22437

  10. Stefan Bürk uploaded patch set 5 01/24/22, 19:55

  11. Stefan Bürk updated the commit message in patch set 6 01/24/22, 19:58

  12. core-ci voted on patch set 5 V +1 01/24/22, 20:10 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22487

  13. core-ci voted on patch set 6 V +1 01/24/22, 20:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/22488

  14. Benni Mack commented on patch set 6 01/25/22, 14:02

    RedirectCacheService.php:61

    we could make this argument optional. This way rebuild() would rebuild all while rebuild(string $limitToSourceHost) would only do this for a single one.

    SlugService.php:135

    we should have a dedicated method $redirectService->rebuildForHosts(array $sourceHosts); to save perf
  15. Stefan Bürk commented on patch set 6 CR -1 01/25/22, 14:33

    will take care later, thanks for review and comments Benni.

    RedirectCacheService.php:61 Reply

    If I get it right, if $sourceHost is empty/null get all unique (distinct) source hosts and iterate over them. If $sourceHost is given, early return/call new method "rebuildForHosts()". Correct ? But that would mean, to put the "host" level back to the resultset, of having this level for empty/unset $sourceHost and if set without (which would be ugly). Do we really need a rebuild all ? Adding this without using this possibility would be superflous, and as the RedirectCacheService is marked internally instances/extension should not use this directly anywy - so where should we use rebuild all than ?

    SlugService.php:135 Reply

    Ack
  16. Benni Mack commented on patch set 6 V +1 CR +1 02/01/22, 21:12

    I think the change is really good. thanks!

    RedirectCacheService.php:61 Reply

    Just checked, I think your current solution is actually just fine :)

    SlugService.php:135 Reply

    let's keep it like this.
  17. Stefan Bürk wrote 02/01/22, 21:14

    Removed Code-Review-1 by Stefan Bürk <stefan@buerk.tech>
  18. Oli Bartsch commented on patch set 6 02/07/22, 17:35

    clusters
    use cases

    DataHandlerCacheFlushingHook.php:35

    No longer unused.

    DataHandlerCacheFlushingHook.php:56

    What would be the use case, for a deleted redirect given to this hook?

    DataHandlerCacheFlushingHook.php:65

    Do we really need this? IMO there should only be one result as we select by UID.

    RedirectCacheService.php:75

    This field can not be NULL, or do I miss sth?

    RedirectService.php:84

    Could just be $redirects now?

    SlugService.php:134

    From reading this method, I would assume this variable is still empty as it get's never assigned any value.
  19. Stefan Bürk commented on patch set 6 02/07/22, 18:05

    DataHandlerCacheFlushingHook.php:56 Reply

    We need to get the source_host of a deleted record, so we can rebuild the cache for that source_host and remove the deleted record from the corresponding cache. Update of the record is done before, so retriving a record for a uid without removing the deleted restriction would not retrieve the as deleted marked redirect. Thus not recaching for that source_host and leaving an deleted redirect in the cache, thus being handled by the middleware on frontentrequests.

    DataHandlerCacheFlushingHook.php:65 Reply

    No, we don't. You are right, should be max one row. Will change this.

    RedirectCacheService.php:75 Reply

    Nope, you are right. This is superflous. Will remove it.

    SlugService.php:134 Reply

    Ack. Using references is not good to see this at a first look from this method. Will change this so this will be directly visible.
  20. Stefan Bürk uploaded patch set 7 02/07/22, 19:16

  21. Stefan Bürk commented on patch set 7 CR -1 02/07/22, 19:17

    still on it

    DataHandlerCacheFlushingHook.php:65 Reply

    Through changing and retesting - this approach to not work here at all. In the hook we only can rebuild all caches. At least with the current hook. For that reason I picked up Benni's original request for a rebuildForHost() and rebuildAll() switch in RedirectCacheService. That's not worser or better than before in this hook, but the frontend middleware and on slug updates this get's done more specificly. Would eventually pick up emproving this hook in a dedicated patch, as I do not have a solid solution for this now. But would not block the benefits of the other changes.
  22. core-ci voted on patch set 7 V -1 02/07/22, 19:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23161

  23. Stefan Bürk uploaded patch set 8 02/07/22, 23:57

  24. core-ci voted on patch set 8 V -1 02/08/22, 00:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23180

  25. core-ci commented on patch set 8 02/08/22, 00:12 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23180

  26. Stefan Bürk uploaded patch set 9 02/08/22, 00:13

  1. core-ci voted on patch set 9 V +1 02/08/22, 00:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23181

  2. Stefan Bürk commented on patch set 9 02/08/22, 00:33

    Reply

    Done

    Reply

    Done

    DataHandlerCacheFlushingHook.php:35 Reply

    Done

    DataHandlerCacheFlushingHook.php:56 Reply

    Done

    DataHandlerCacheFlushingHook.php:65 Reply

    For doing the requested changed from Oliver, it was clear that we were near to a regression. Point is, that we need to recache forl "old" and "new" source_host, if source_host is changed for an update. Ex. "domain.tld" to "*" etc. This was not covered. And could not be correctly determined by simply fetching the updated record from the database. Thus I have done following: * use history records to get old/new source_hosts (update) data operation. * for cmd map operation, do database lookup as there is no history record present and we can safely use the retrieved record from the database * pick up old-comment from Benni and added a rebuildAll() into RedirectCacheService and used to rebuild all caches (as before) - should not be needed, but better have a safety net here before running into an regression for it. Still, not covering the fact that this hook get's called multiple times if multiple records gets changed using DataHandler. Maybe we'll find a way in a dedicated patch to mitigate this or do not need caching in the future anymore.

    RedirectCacheService.php:75 Reply

    Done

    RedirectService.php:84 Reply

    Done

    SlugService.php:134 Reply

    Done
  3. Stefan Bürk rebased onto patch set 10 02/10/22, 18:41

  4. core-ci voted on patch set 10 V +1 02/10/22, 18:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23347

  5. Oli Bartsch commented on patch set 10 02/14/22, 10:43

    DataHandlerCacheFlushingHook.php:60

    $sourceHost === []

    SlugService.php:211

    What do you think about returning the created record and not just a "random" single field?

    SlugService.php:214

    Maybe add a PHPDoc here to explain the return value, since it is not obvious that "checkSubPages" will return an array of "source hosts".
  6. Stefan Bürk commented on patch set 10 02/15/22, 15:50

    SlugService.php:211 Reply

    Generally I do not like to return unneeded data, but in this case I would say that's more conveniant to return the full record, even if we currently need the source_host. Will change this.
  7. Stefan Bürk uploaded patch set 11 02/15/22, 15:54

  8. core-ci voted on patch set 11 V -1 02/15/22, 16:04 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23634

  9. Stefan Bürk uploaded patch set 12 02/15/22, 16:21

  10. core-ci voted on patch set 12 V +1 02/15/22, 16:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23640

  11. Stefan Bürk commented on patch set 12 02/15/22, 17:03

    DataHandlerCacheFlushingHook.php:60 Reply

    Done

    DataHandlerCacheFlushingHook.php:65 Reply

    Done

    SlugService.php:211 Reply

    Done

    SlugService.php:214 Reply

    Done
  12. Oli Bartsch uploaded patch set 13 02/15/22, 23:41

  13. Oli Bartsch uploaded patch set 14 02/15/22, 23:50

  14. core-ci voted on patch set 13 V -1 02/15/22, 23:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23694

  15. core-ci voted on patch set 14 V +1 02/15/22, 23:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23696

  16. Stefan Bürk uploaded patch set 15 02/16/22, 00:13

  17. core-ci voted on patch set 15 V +1 02/16/22, 00:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23698

  18. Oli Bartsch uploaded patch set 16 02/16/22, 14:43

  19. Oli Bartsch voted on patch set 16 V +1 CR +1 02/16/22, 14:49

  20. core-ci voted on patch set 16 V +1 02/16/22, 14:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/23728

  21. Stefan Bürk commented on patch set 16 CR +1 02/16/22, 14:54

    semi +1 cr&v on changes from Oliver, rechecked changes. Thanks for the help, Oliver.
  22. Benni Mack voted on patch set 16 V +1 CR +1 02/17/22, 12:59

  23. Benni Mack voted on patch set 16 V +2 CR +2 02/17/22, 14:00

  24. Benni Mack merged patch set 17 02/17/22, 14:00

  25. Oliver Hader commented on patch set 17 05/09/22, 11:09

    RedirectCacheService.php:82 Unresolved

    `in` can probably enhanced to `eq` - since we're not dealing with multiple values in `$sourceHost` here