Review main Merged

86296[BUGFIX] Prevent race condition while writing file caches

Benjamin Franzke CR +2 V +2 Normal +36 −29 updated 01/07/25, 08:13

Commit message

Whenever a cache entry is written via the cache file backends,
two race conditions could previously happen when a parallel flush
is in action:

a) The temporary file is trying to be written to a folder that is
   currently not available, because it has been moved, but not
   recreated yet. The time frame for this race condition should
   have been quite slow, but it's still the error we saw most of
   the times, which likely happened often due to stale NFS caches.

b) The temporary cache (that is created in a file named `{random}.temp`)
   could be written to the cache directory, but is (re)moved to another
   directory because a parallel `flush()` moves the containing folder to
   a temporary folder, in order to delete it afterwards.
   That means the temporary file can then still be written to (as long
   as the folder is still being deleted, the kernel file descriptor that
   PHP writes to is still valid), but the subsequent `rename` operation
   will fail because the temporary file is tried to be renamed (moved)
   from the "old" location (within the original cache folder), but is no
   longer stored there.

In order to prevent both race conditions, cache folders are no
longer removed. Only the containing cache files are deleted.

The temporary cache files are now created in a separate tmp-files
subfolder, in order to not be removed by a parallel flush operation.

Review

  • Code review CR +2 Done
    • Benjamin Franzke +2
    • Markus Klein +1
    • Garvin Hicking +1
    • Anja Leichsenring +1
    • Stefan Bürk +1
    • Andreas Kienast +1
  • Verified V +2 Done
    • Benjamin Franzke +2
    • Garvin Hicking +1
    • Anja Leichsenring +1
    • Stefan Bürk +1
    • Andreas Kienast +1
    • core-ci +1 CI

Issues 1

Files 2

Review log 2126

Show 1 earlier messages
  1. Benjamin Franzke uploaded patch set 1 09/24/24, 12:30

  1. core-ci voted on patch set 1 V -1 09/24/24, 12:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/66495

  2. Mathias Brodala commented on patch set 1 09/24/24, 12:44

    In any case a huge 👍 for the commit message which clearly describes the issue at hand.

    SimpleFileBackend.php:194

    Are we sure this ends with a slash? Or is either case ("foo/tmp/" & "footmp/") intentionally OK?

    SimpleFileBackend.php:248

    As mentioned in the other change it may be beneficial to check if https://www.php.net/error_get_last can provide a few more details. But I'd be fine to do this separately to keep this one small.

    SimpleFileBackend.php:321

    Should we also check for success here?
  3. Benjamin Franzke uploaded patch set 2 V -1 09/24/24, 12:54

  4. core-ci voted on patch set 2 V +1 09/24/24, 13:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/66496

  5. Benjamin Franzke uploaded patch set 3 V +1 09/24/24, 15:21

  6. Benjamin Franzke commented on patch set 3 09/24/24, 15:21

    SimpleFileBackend.php:194 Reply

    I was also stumbling about that, but the entire class presumes it has a trailing slash and that is done in `setCache` (and tested in functionals). This is "ok" (although I don't like it either, but did it to keep the code aligned)

    SimpleFileBackend.php:321 Reply

    Done
  7. core-ci voted on patch set 3 V +1 09/24/24, 15:31 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/66507

  8. Benjamin Franzke uploaded patch set 4 V +1 09/25/24, 04:44

  9. Benjamin Franzke commented on patch set 4 09/25/24, 04:44

    SimpleFileBackend.php:248 Reply

    Not possible as far as I see, because we have a custom error handler, we can not use `error_get_last()`: can be verified via: ``` php -r "@unlink('/root/noaccess');var_dump(error_get_last());error_clear_last();set_error_handler(fn() => null);unlink('/root/noaccess');var_dump(error_get_last());" ``` Which delivers null for the same mistake, once an error handler is attached: ``` array(4) { ["type"]=> int(2) ["message"]=> string(41) "unlink(/root/noaccess): Permission denied" ["file"]=> string(17) "Command line code" ["line"]=> int(1) } NULL `
  10. core-ci voted on patch set 4 V +1 09/25/24, 04:49 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/66535

  11. Benni Mack updated the commit message in patch set 5 V +1 10/18/24, 13:55

  12. Benjamin Franzke rebased onto patch set 6 11/04/24, 10:05

  13. core-ci voted on patch set 6 V +1 11/04/24, 10:12 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69027

  14. Markus Klein commented on patch set 6 CR +1 11/05/24, 07:31

    Reads good to me. In order to really rule out race conditions, we would need to employ an exclusive lock.
  15. Anja Leichsenring voted on patch set 6 CR +1 V +1 11/08/24, 10:01

  16. Markus Klein commented on patch set 6 11/10/24, 18:22

    FileBackend.php:71

    Just a thought: It was "isDot" before: Do we need to handle hidden files here too? This was covered before, but not anymore.
  17. Benjamin Franzke commented on patch set 6 12/19/24, 09:29

    FileBackend.php:71 Reply

    I don't think so, as we do not write hidden files to this directory.
  18. Markus Klein commented on patch set 6 12/19/24, 13:16

    FileBackend.php:71 Reply

    I mean, what if a `.something` file is in the directory? (very unlikely) It was skipped before, but now isn't, right?
  19. Benjamin Franzke commented on patch set 6 12/19/24, 13:40

    FileBackend.php:71 Reply

    yes, understood – but if something else is creating that file (for whatever reason) I see no reason in having to delete it(?)
  20. Markus Klein commented on patch set 6 12/19/24, 18:04

    FileBackend.php:71 Reply

    Acknowledged
  21. Garvin Hicking commented on patch set 6 CR +1 V +1 12/19/24, 18:42

    I ran my dev env with ddev&mutagen for some days with this applied, and believe it fixed sporadic issues with the file cache. Couldn't pinpoint it before, and it never occured without mutagen.
  22. Andreas Kienast voted on patch set 6 CR +1 V +1 12/20/24, 13:57

  23. Stefan Bürk voted on patch set 6 CR +1 V +1 01/03/25, 14:14

  24. Benjamin Franzke voted on patch set 6 CR +2 V +2 01/07/25, 08:13

  25. Benjamin Franzke merged patch set 7 01/07/25, 08:13