Issue Bug Needs Feedback Orphaned

#105555TypoScript caching locks other page calls exklusive

Andreas Hager 11/07/24, 12:17 updated 05/04/26, 15:14

Description

The TypoScript frontend controller locks complete website on each call exclusivly.

We have the following setup:
- Load Balancer + Varnish
- 3 Webserver with TYPO3 on each Node
- NFS Share for shared files
- Redis Server

Due to NFS share we cannot use file locking strategy – we use Redis locking strategy (https://github.com/b13/distributed-locks).
We have approximately 7-10 Requests per second and 1.3 million pages in TYPO3. Many of them are not cached, so the page cache, especially the TypoScript cache has to be generated new on different calls.

The aquire lock is on 3 different places in the code:
- https://github.com/TYPO3/typo3/blob/v12.4.22/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L1402
- https://github.com/TYPO3/typo3/blob/v12.4.22/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L1428
- https://github.com/TYPO3/typo3/blob/v12.4.22/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php#L1432

The locking here causes the complete T3 to wait for the lock to be release (the lock name is ‘pages’). So if there are 7 calls per second and for 10 seconds -> 70 calls has to be handled. The locking part takes for example 1 second to lock/unlock -> after 10 seconds only 10 requests of 70 are handled and the PHP FPM process starts to spawn until the limit is reached.

In the ResourcesMutex an exclusive and non-block lock is done for the key ‘pages’ (https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L87-L89) which causes all other pages to wait until a random page is finished.

I would expect that only this specific page is locked instead of all pages are locked (with the hash included).

What if the the code is adapted to:

$this->lock->acquireLock('pages_' . $this->newHash, $this->newHash);

Then the lock is acquired only per specific page instead that everything is locked.

The same ResourceMutex is used in T3 v13 – so the problem should be the same.

How to reproduce?

1. Add a sleep(40); before line 90 (https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L90)
2. Call one random page without cache.
3. Now call another page without cache and the second call will wait until the first call ist finished :)

If you need further information let me now!

Thanks Andi

Reviews 1

Fix this issue

Start the commit message of your patch with this. Add what the patch does, and the versions it goes to in "Releases:". The changes above were merged or abandoned: read them before you start. How to create a patch

Commit message
[BUGFIX] TypoScript caching locks other page calls exklusive Resolves: #105555 Releases: main

Files 1

History 89

  1. Marcel Macasso changed attached acquire-lock-for-pages-with-hash.patch <span data-processor="relativetime" data-value="2024-11-12T13:38:32+00:00" > 11/12/24, 13:38 </span>

  2. Sybille Peters commented <span data-processor="relativetime" data-value="2024-11-12T18:49:42+00:00" > 11/12/24, 18:49 </span>

    Hello, just my 2c.

    In my experience, this might get more attention if you:

  3. Oliver Pfaff commented <span data-processor="relativetime" data-value="2025-09-11T20:29:18+00:00" > 09/11/25, 20:29 </span>

    We had similar phenomena in a project. For unknown reasons, the server recorded a rapid increase in PHP child processes at different times, until it reached its maximum. This caused the server to go down. The server recovered after 40-50 minutes on its own, and PHP resets could also counteract it. The next spike repeated itself hours or days later, without a consistent pattern.

    Our setup is far less refined than Andreas’. It’s a VM running only this project, with 16 dedicated vCPUs and 64GB RAM. The monitoring reports 25 to 35 req/s. CPU and RAM always show plenty of free capacity.
    The TYPO3 system (V12) serves 16 different websites.

    My suspicion was that I/O blocking was occurring in the PHP processes. For most of our requests, external APIs are involved. These connections were checked and had timeouts configured. Blocking is, of course, possible with database queries, but the monitoring didn’t show any real problematic queries. We logged slow requests on the web server, which pointed to ResourceMutex.php. Cache and DB locks were considered from the very beginning when analyzing the problem. Based on the clues, I ended up here and took a closer look at the locking and the patch.

    For what i see, the locking works as intended. Props to this clever integration. I miss the days of “page is being generated” ;).

    It is correct that a general access lock is generated for pages.
    https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L80
    This lock ensures that the check for the specific lock(page chash) for the page, which is generated from a hash of various page-specific parameters,
    https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L81-L83 can only be checked exclusively by one process.
    However, the access lock is released only a few lines later after the check.
    Here https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L97 when the specific page lock is not free, and thus another process is currently generating the cache for that page.
    Here https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L105 when the current process has a clear path.
    Only in this very short period would it not be possible for another page to build a cache. I did not measure the exact time span, and we are using the FileLockStrategy. The time span could possibly be larger when querying through Redis (possibly due to latency or accessibility), but I’d be surprised. Maybe with suboptimal Redis settings or state?

    What I don’t fully understand is, if two processes really get in each other’s way in the short possible time period at https://github.com/TYPO3/typo3/blob/main/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L87-L89 , wouldn’t the exception be triggered and the one process wouldn’t recover? Or am I overlooking a catch? Or is there another twist?

    Our problem and solution

    For us, the problem that led to the child spikes was a very unfortunate implementation. An Extbase action that was supposed to be cached had the cache disabled through TSFE when certain GET parameters were present. This led TYPO3 to try to build a cache and use the locking, but in our case, the requests only differed through GET parameters unknown to the system (these are ignored when generating the cHash to avoid cache flooding), so the generated hash was always the same, even though the GET parameter values were different. When many requests with these GET parameters occurred simultaneously, they were processed sequentially due to the locking, and each process had to generate the page completely when it got its turn, since no cache entry was saved due to the cache being disabled via TSFE.
    The solution is to include the GET parameters in the cHash generation, for which there’s a hook in V12 ($GLOBALS[‘TYPO3_CONF_VARS’][‘SC_OPTIONS’][‘tslib/class.tslib_fe.php’][‘createHashBase’]). The cache flooding risk must then be handled onyour own in the action itself. Alternatively, a non-cached Extbase action can be used (in which case the locking does not apply), optionally with custom cache management. That would have been the clean implementation for our case.

  4. Benni Mack commented and changed status Under Review Needs Feedback <span data-processor="relativetime" data-value="2026-05-04T13:24:00+00:00" > 05/04/26, 13:24 </span>

    Could this still happen in v13?

  5. Andreas Hager commented <span data-processor="relativetime" data-value="2026-05-04T14:28:58+00:00" > 05/04/26, 14:28 </span>

    I think yes :-)

    https://github.com/TYPO3/typo3/blob/v13.4.28/typo3/sysext/frontend/Classes/Middleware/PrepareTypoScriptFrontendRendering.php (Line 88 + 100)

    aquires a lock of ‘pages’

    https://github.com/TYPO3/typo3/blob/v13.4.28/typo3/sysext/core/Classes/Locking/ResourceMutex.php#L87

    Here the exclusive access lock on key ‘pages’ is done, when the access locking takes a little bit of time (even when the access lock is only a short period of time) with a multi server setup and Redis locking strategy (because file locking does not work for multi server setup).

  6. Oliver Pfaff commented <span data-processor="relativetime" data-value="2026-05-04T15:05:32+00:00" > 05/04/26, 15:05 </span>

    From my perspective, it’s necessary to lock pages for a few milliseconds. If there isn’t an exclusive lock on pages for a few milliseconds, two processes could generate the specific page defined by the $key at the same time. That’s exactly what the implementation aims to prevent: ensuring that only one process actually generates a page with a specific and unique $key (cHash) and writes it to the cache. Not multiple processes running in parallel that end up overwriting each other.
    However, the exclusive lock on pages should be released after a few milliseconds. If Redis takes nearly 1 second or a few hundred milliseconds here, then that is obviously a problem, which could actually cause child processes to overflow in high-traffic/high-performance systems.

  7. Andreas Hager commented <span data-processor="relativetime" data-value="2026-05-04T15:14:57+00:00" > 05/04/26, 15:14 </span>

    Thats true, somehow you have to prevent other processes to build the same cache multiple times.

    But I think that you don’t need an exclusive lock at the key ‘pages’ because if the cache for page with UID 1 is generated, the page with UID 2 could be generated at the same time. Maybe there should be two exklusive locks:
    - pages_1
    - pages_2

    or even with the complete page cache hash instead of locking ‘pages’.
    So every unique page thats get written to the cache should have its own locking namespace :-)