Review main Merged

79146[FEATURE] Implement simple log rotation

Andreas Kienast CR +2 V +2 Hard +495 −0 updated 07/10/23, 07:51

Commit message

TYPO3 logs tend to grow over time if not manually cleaned on a regular
basis, potentially leading to full disks. Also, reading its contents
may be hard when several weeks of log entries are printed as a wall of
text.

To circumvent such issues, established tools like `logrotate` are
available for a long time already. However, TYPO3 may be installed on a
hosting environment where `logrotate` is not available and cannot be
installed by the customer. To cover such cases, a simple log rotation
approach is implemented.

A new file writer `\TYPO3\CMS\Core\Log\Writer\RotatingFileWriter` is
added that extends the already existing `FileWriter` class. The
`RotatingFileWriter` accepts two options:

The interval how often log files should be rotated can be configured
with any case of `\TYPO3\CMS\Core\Log\Writer\Enum\Interval`,
`Interval::DAILY` is the default. It's also possible to configure how
many log files are retained, where `5` is the default.

Review

  • Code review CR +2 Done
    • Andreas Kienast +2
    • Jochen Roth +1
    • Susi Moog +1
  • Verified V +2 Done
    • Andreas Kienast +2
    • Jochen Roth +1
    • Susi Moog +1
    • core-ci +1 CI

Issues 1

Files 4

Review log 3950

Show 1725 earlier messages
  1. Andreas Kienast uploaded patch set 1 05/25/23, 15:51

  2. core-ci voted on patch set 1 V +1 05/25/23, 15:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42021

  3. Andreas Kienast uploaded patch set 2 V +1 05/25/23, 16:21

  4. core-ci voted on patch set 2 V +1 05/25/23, 16:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42022

  5. Andreas Kienast uploaded patch set 3 V +1 05/26/23, 13:45

  6. Andreas Kienast uploaded patch set 4 05/26/23, 13:53

  7. core-ci voted on patch set 3 V -1 05/26/23, 13:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42050

  8. core-ci voted on patch set 4 V -1 05/26/23, 14:05 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42051

  9. Andreas Kienast uploaded patch set 5 V -1 05/28/23, 10:50

  10. core-ci voted on patch set 5 V +1 05/28/23, 10:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42075

  11. Mathias Brodala commented on patch set 5 05/30/23, 07:43

    IMO this should be done as a CLI command / scheduler task instead. Benefits: - we can use existing scheduling configuration - we avoid timing issues / race conditions in case of rotating large log files at runtime
  12. Andreas Kienast commented on patch set 5 06/02/23, 06:32

    Reply

    I'm not sure how this should work if I want different rotation cycles for different log files.
  13. Mathias Brodala commented on patch set 5 06/05/23, 07:16

    Reply Unresolved

    Create multiple tasks with the same command but different config for example.
  14. Anja Leichsenring commented on patch set 5 06/13/23, 04:51

    I like that dependency-less approach. While a scheduler or cron task also works, it relies on a working cron triggering it. Some integrator also has to set it up. This feature will work without any triggering, only once being configured, and can be forgotten forever. So +1 from me, while not opposing a potential additional CLI based task, too.
  15. Anja Leichsenring voted on patch set 5 CR +1 06/13/23, 04:51

  16. Andreas Kienast commented on patch set 5 06/13/23, 06:00

    Reply

    I'd like to get more feedback on this, please. If the vast majority opts for a scheduler task, I can of course change the patch.
  17. Andreas Kienast uploaded patch set 6 CR +1 V +1 06/16/23, 08:00

  18. Andreas Kienast uploaded patch set 7 06/16/23, 08:03

  19. core-ci voted on patch set 6 V -1 06/16/23, 08:16 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42721

  20. core-ci voted on patch set 7 V -1 06/16/23, 08:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42723

  21. Andreas Kienast commented on patch set 7 V -1 06/16/23, 08:34

    The patch breaks the Install Tool (for me) /o\
  22. Andreas Kienast uploaded patch set 8 V -1 06/16/23, 09:18

  23. core-ci voted on patch set 8 V +1 06/16/23, 09:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42729

  24. Markus Klein commented on patch set 8 06/16/23, 14:31

    RotatingFileWriter.php:42

    Any specific reason to use a ResourceMutex here instead of just a simple lock?
  25. Andreas Kienast commented on patch set 8 06/16/23, 17:13

    RotatingFileWriter.php:42 Reply

    I honestly didn‘t fully understand how I should configure the lock (exclusive, shared, etc.) and didn‘t find any usage in Core but the ResourceMutex.
  1. Markus Klein commented on patch set 8 06/16/23, 17:46

    RotatingFileWriter.php:42 Reply

    Alright. I'll do a take on that one then, if you don't mind. Basically: If you need to have a look for an potentially endless number of resources (like pages), then use ResourceMutex. Otherwise a simple exclusive lock is sufficient.
  2. Markus Klein uploaded patch set 9 V +1 06/16/23, 17:56

  3. Markus Klein commented on patch set 9 06/16/23, 17:57

    RotatingFileWriter.php:42 Reply

    Done
  4. core-ci voted on patch set 9 V -1 06/16/23, 18:07 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/42748

  5. Christoph Lehmann commented on patch set 9 06/16/23, 18:42

    Pretty nice feature! I prefer to rotate on the maximum filesize rather than on interval. At some hosters, log files don't grow above 2GB and new lines get lost, so from my point of view the filesize should be default reason for logrotation and the interval cooouuld be an option. https://github.com/cesargb/php-log-rotation looks like a nice source. By leaving away the interval option we don't need a cli task doing the rotation. Furthermore i would like the feature in the TYPO3 defaults FileWriter, not a new one, that needs to be configured manually (to have strong defaults). The minimum/maximum filesize and maximum number of could be defined in DefaultConfiguration.php and might be overwritten in LogWriter configuration.
  6. Christoph Lehmann commented on patch set 9 06/16/23, 18:49

    Reply

    On the other hand.. the interval is also nice, but then please keep .log as the file extension and not the date.
  7. Markus Klein commented on patch set 9 06/17/23, 07:30

    Reply

    I think Andy used Linux's logrotate as template, which appends the date
  8. Andreas Kienast commented on patch set 9 06/29/23, 08:06

    Reply

    Indeed. Also, with the date I'm able to determine when the last rotation happened without the need to store the timestamp somewhere else. The filesize aspect is something I'd like to introduce in a separate patch if really necessary. Also, this is just a bare-minimum implementation which shall not replace `logrotate`.
  9. Andreas Kienast commented on patch set 9 CR -1 06/29/23, 08:13

    RotatingFileWriter.php:83

    phpstan reports this line with "Right side of && is always true." which is true as the outer condition evaluated to `true` before. The issue is there's no code in between that alters `$this->nextRotation` or `$this->lastRotation`.
  10. Markus Klein commented on patch set 9 06/29/23, 08:18

    RotatingFileWriter.php:83 Reply

    Right, I missed that needsRotation does not update it's data. We need to update the lastRotation/nextRotation information in this method
  11. Andreas Kienast uploaded patch set 10 CR -1 V -1 06/29/23, 12:47

  12. Andreas Kienast commented on patch set 10 06/29/23, 12:47

    RotatingFileWriter.php:83 Reply

    Done
  13. core-ci voted on patch set 10 V +1 06/29/23, 12:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43112

  14. Markus Klein voted on patch set 10 CR +1 07/03/23, 21:40

  15. Susi Moog voted on patch set 10 V +1 CR +1 07/07/23, 12:29

  16. Jochen Roth voted on patch set 10 V +1 CR +1 07/07/23, 22:33

  17. Oliver Klee commented on patch set 10 07/08/23, 07:29

    Interval.php:3

    Please let's make this file strict.

    Interval.php:25

    Can we have a simple test for this method?
  18. Andreas Kienast commented on patch set 10 07/10/23, 05:48

    Interval.php:3 Reply

    Ack

    Interval.php:25 Reply

    A test for a getter method with basically no logic? If something breaks *here*, other tests will fail anyway.
  19. Andreas Kienast uploaded patch set 11 CR +1 V +1 07/10/23, 05:49

  20. core-ci voted on patch set 11 V +1 07/10/23, 05:56 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43965

  21. Susi Moog voted on patch set 11 V +1 CR +1 07/10/23, 07:16

  22. Jochen Roth voted on patch set 11 V +1 CR +1 07/10/23, 07:33

  23. Oliver Klee commented on patch set 11 07/10/23, 07:37

    Interval.php:25 Reply

    Yes, that's true. The idea of having tests for more classes in a call stack is that you won't only notice _that_ something is broken, but it will also help locate you _what_ is broken (which can be a big time-saver for debugging).
  24. Andreas Kienast voted on patch set 11 V +2 CR +2 07/10/23, 07:51

  25. Andreas Kienast merged patch set 12 07/10/23, 07:51