Review 12.4 Abandoned

86981[BUGFIX] Replace exclusive page lock with lock per page

Andreas Hager CR -1 V -1 Easy +3 −3 updated 05/05/26, 09:02

Commit message

Review

  • Code review CR -1
    • Benni Mack -1
  • Verified V -1
    • Markus Klein -1
    • core-ci -1 CI

Issues 1

Files 1

Review log 2122

  1. Andreas Hager uploaded patch set 1 11/13/24, 12:45

  2. Markus Klein commented on patch set 1 V -1 11/13/24, 12:59

    The number of locks scales with the number of pages in your instance. If you are using semaphore locking and the OS limit is less than number of pages you will literally kill the server. We tried that back then when I implemented the locks.
  3. Andreas Hager commented on patch set 1 11/13/24, 13:15

    Reply

    Thanks for your explanation 😊 Thats right but in our case we use Redis as locking strategy and so we have a huge amount of semaphore locking. Maybe this has to defined somehow by locking strategy? Or maybe a limit for concurrent locks should be implemented? But using 1 exclusive lock for all page generation with multiple concurrent calls on different pages will slow down TYPO3 and spawn FPM processes that have to wait until page generation is finished.
  4. core-ci voted on patch set 1 V -1 11/13/24, 13:53 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69396

  5. Markus Klein commented on patch set 1 11/13/24, 14:31

    Reply

    > Or maybe a limit for concurrent locks should be implemented? You'd need an exclusive lock in order to determine this. > But using 1 exclusive lock for all page generation with multiple concurrent calls ... Please check the internals of ResourceMutex. This is actually a combination of 2 nested locks. 1 exclusive, which is held for an extremely short time and 1 per page, which is NON-BLOCKING. You should not see any FPM process waiting for anything, unless your lock strategy (file-based per default) does not support non-blocking locks. Usually happens, if var/locks/ resides on a non-atomic NFS file system.
  6. Sybille Peters commented on patch set 1 11/17/24, 14:17

    Usually, you should push to main branch. The fix will then be backported if appropriate and the Release is in "Releases". Only push directly to a lower branch if fix should be only for this particular version. Example: (in commit message): Releases: main, 13.4, 12.4
  7. Garvin Hicking commented on patch set 1 12/31/24, 01:06

    Any thoughts how/if to proceed with this? Markus' note to me reads is if this is more a setup issue and probably not an optimization that should be made on the TYPO3 core side?
  8. Garvin Hicking commented on patch set 1 12/31/24, 01:06

    Reply

    .
  9. Sybille Peters wrote 12/31/24, 12:43

    Removed cc Sybille Peters.
  10. Benni Mack commented on patch set 1 CR -1 03/22/25, 14:18

    Reply Unresolved

    Any feedback @a.hager@limesoda.com ? Otherwise I agree with Markus, Haven't had any problems in this area (both NFS and regular SSD)
  11. Andreas Hager commented on patch set 1 05/26/25, 11:44

    Reply

    Sorry for the late response, I think this is still an TYPO3 issue and not a setup issue 😊 I wrote a detailed description here (https://forge.typo3.org/issues/105555) how you can reprocude the problem. If you have a lot of traffic with a lot of sites in TYPO3 and you flush the cache, the complete frontend is locked on each request, until the cache for this single webpage is built. All other pages cannot create their cache until this page is finished. Maybe we can find a different solution to this problem? The same problem appears in the PrepareTypoScriptFrontendRendering.php middleware in the main branch.
  12. Markus Klein commented on patch set 1 05/26/25, 12:21

    Reply

    I'll read your description in the ticket again, but currently I still don't get this > the complete frontend is locked on each request, until the cache for this single webpage is built. This should simply not be the case. That's the whole point of the existing locking solution. This ONLY happens, if your locking does not support shared locks.
  13. Andreas Hager commented on patch set 1 05/26/25, 12:55

    Reply

    Maybe I misunderstand something, but lets have a look at the frontend cache generation in TYPO3 v14 😊 In this middleware (https://github.com/TYPO3/typo3/blob/main/typo3/sysext/frontend/Classes/Middleware/PrepareTypoScriptFrontendRendering.php#L100) on Line 88 or Line 100 "acquireLock('pages', $pageCacheIdentifier)" is called. If we have a closer look in the ResourceMutex (https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L80) on Line 80 the accessLocks creates a locker with key 'pages'. The method createLocker creates an exclusive lock on the key 'pages' (https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/LockFactory.php#L70) because the second argument is ommited. Now all other requests that try to build the cache from any other webpage for this TYPO3 installation has to wait because there is an exclusive lock on key 'pages'. If the generation of the current page takes for example 10 seconds all other webpages have to wait 10 seconds until the next webpage can be processed. Hope I explained it a little bit better 😊
  14. Andreas Hager commented on patch set 1 06/02/25, 08:24

    Reply

    Do you need further information? Or could you reproduce the problem? 😊
  15. Markus Klein commented on patch set 1 11/10/25, 12:26

    Reply

    Nope, the global pages lock is only held for a very short time to retrieve the actual worker lock. https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L105
  16. Benni Mack abandoned the change 05/04/26, 13:23

    support for 12.4 version in the free version has been dropped.
  17. Andreas Hager commented on patch set 1 05/05/26, 06:22

    Reply

    That is true but if you use for example Redis locking strategy (because you have multiple webservers) and your Redis server takes a little bit longer to lock/release than this short time gets bigger and bigger and the PHP processes starts to spawn and hang 😊
  18. Andreas Hager commented on patch set 1 05/05/26, 06:24

    Any chance that this will come to TYPO3 v13 or v14? 😊
  19. Benni Mack commented on patch set 1 05/05/26, 06:37

    Reply Unresolved

    sure, but you'd need to push a patch to the main branch (not v12 branch)
  20. Markus Klein commented on patch set 1 05/05/26, 06:59

    Reply

    Well, that's a big conceptional question then. This is a totally different use case. If you distribute caches a lot more things come up. Our take on this: Keep as many caches as local as possible. Aka: We usually keep rootline and page caches local to the webserver.
  21. Andreas Hager commented on patch set 1 05/05/26, 07:04

    Reply

    But with a multi server setup, that is not possible because we have a decicated Database and Redis server.
  22. Markus Klein commented on patch set 1 05/05/26, 09:02

    Reply

    Why not? It's totally possible to serve the caches from a separate local mysql or from a local redis server. Besides that, there are so many ways multiserver/multi-head setups are possible that a general solution is simply not viable. So Core must support the 90% case. Another note: even if you have a central DB+Redis and shared locks via Redis, you still face the same issue that the core's solution actually solves: Do not generate the same page more than once at the same time. There is no point in letting 100 server generate the same page at the same time. I do not fully understand your bottleneck though. Since the locks are only requested if we have a cache-miss. Yes, all servers and their processes are in waiting time until 1 server finishes generating the page, but then things should be independent again.