Review main Abandoned

48174[WIP][FEATURE] Replace logging framework with monolog

Susi Moog CR 0 V 0 Nightmare +678 −3402 updated 02/28/20, 09:47

Commit message

This patch introduces the requirement for monolog/monolog
which will replace the former logging framework of TYPO3.

Both the former framework and monolog are PSR-3 compatible,
thus there is no change in actual log calls. However, there
is a change regarding the configuration of monolog in TYPO3.

The goal is to embrace the idea of channels and looking at
the configuration like symfony does. To understand why things
are configured the way they are, one has to understand the
concept of monolog.

There are loggers, channels, handlers, processors and formatters.
While this commit message can't go into details about all these
things, it will roughly explain how they work together.

A logger is the instance that represents PSR-3, along with the
callable log methods (error, warning, debug and such).

Each logger represents a channel. But a channel is not the name
of a logger as different loggers can represent (write into) the
same channel. A channel is like a category. While most log entries
are fine to be written into a default channel, a failed login
attempt should be written into a security channel. A channel does
not define where a log is written to, it only categorizes the
log entries.

Then there are handlers. They define how and where a log entry
is written (to). The default handler of TYPO3 is a StreamHandler
that writes log files to typo3temp/var/logs/typo3.log.
While a handler defines how and where a log entry is written (to),
the so to subscribed channels define which log entries are handles
by the handler. TYPO3 defines a default channel which – along with
the default StreamHandler - is some kind of a fallback logging
strategy.

Processors are actually just callbacks that work on the log entry
and alter it. An example is a uid processor, that adds a unique
id to all log entries.

Formatters are objects that define how the log entry is formatted.
The default formatter is a line formatter, which is suitable for
logging to files.

As the main concepts of monolog are embraced, the official docs
of monolog are a must read to fully understand the possibilities
and why the configuration works the way it does.

About the configuration:
$GLOBALS['TYPO3_CONF_VARS']['LOG'] is still the entry point for
the configuration of the logging framework. However, the available
keys have changed.

The following keys do exist:
$GLOBALS['TYPO3_CONF_VARS']['LOG']['handlers']
$GLOBALS['TYPO3_CONF_VARS']['LOG']['processors']
$GLOBALS['TYPO3_CONF_VARS']['LOG']['loggers']

$GLOBALS['TYPO3_CONF_VARS']['LOG']['handlers'] defines all
available handlers in the installation and the configration
does no differ much from the one of the former logging framework.

$GLOBALS['TYPO3_CONF_VARS']['LOG']['processors'] are globally
availbale log entry processors. As a default there are no global
processors defined but it's likely to change as global processors
are a great way to add extra information to the log entry.
Backend user id, frontend user id, unique request id and many more
are examples of additional data that might be added globally via
processors.

$GLOBALS['TYPO3_CONF_VARS']['LOG']['loggers'] is a simple mapping
of class name to channel name. By default, all classes that do not
have a mapping here, will get a logger instance that logs to the
default channel. This configuration is a very simple represen-
tation of the dependency injection configuration of symfony.

As TYPO3 does not have such a sophisticated dependency injection
and dependency configuration like symfony, the possibilites are
rather limited at the moment. The goal is to define loggers by
extension or namespace in the future.

Although this is not possible at the moment, the implementation
of monolog still drastically imporves the possibilites to
properly log what's going on and especially where to log to.

todo:
- documentation
- config migration wizard

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 54

Show 34 more files

Review log 4349

Show 2224 earlier messages
  1. Susi Moog uploaded patch set 1 05/15/16, 13:17

  2. Susi Moog uploaded patch set 2 05/16/16, 10:58

  3. Susi Moog uploaded patch set 3 05/16/16, 11:01

  4. Susi Moog uploaded patch set 4 05/16/16, 14:21

  5. Susi Moog commented on patch set 4 05/16/16, 14:25

    LogManager.php:68

    my mistake, will fix that
  6. Georg Ringer commented on patch set 4 05/16/16, 17:52

    AboutController.php:18

    are those used?

    LogManager.php:209

    no hsc needed, this must be done when rendering the exception

    Feature-12345-ReplaceLoggingWithMonolog.rst:49

    this would require a breaking rst IMO

    ImportExportUtility.php:66

    please replace those with the according class of monolog instead of removing it
  7. Susi Moog commented on patch set 4 05/17/16, 05:23

    AboutController.php:18 Reply

    good catch, not anymore - tested with this class.

    ImportExportUtility.php:66 Reply

    removed them because we now have returnType completion for makeInstance and getLogger always returns \Monolog\Logger which the IDE knows, so there is no need for them anymore.
  8. Susi Moog commented on patch set 4 05/17/16, 05:23

    Feature-12345-ReplaceLoggingWithMonolog.rst:49 Reply

    Additionally or instead of this one? I was not sure.
  9. Steffen Müller commented on patch set 4 05/17/16, 07:04

    Thank you Susi for picking this task. We should use Psr\Log\LoggerInterface instead of Monolog\Logger at those places where we create logger instances. This is already the case for example in https://github.com/TYPO3/TYPO3.CMS/blob/c78be12c03c919273aa2332fd080fd0b4257a2d5/typo3/sysext/fluid/Classes/Core/ViewHelper/AbstractViewHelper.php#L221
  10. Steffen Müller uploaded patch set 5 05/17/16, 07:29

  11. Steffen Müller commented on patch set 5 05/17/16, 07:47

    A drawback I see with the current patchset is that the requestId is dropped, but it IMHO should be kept. It is the key to identify all log messages of a unique call.

    DefaultConfiguration.php:1150

    In current releases we append a random has to the default log file name for security-by-obscurity reasons. We should not drop that feature. https://github.com/TYPO3/TYPO3.CMS/commit/97949799ac7477ba69fc6ca5790219744d536014
  12. Steffen Müller commented on patch set 5 05/17/16, 07:57

    Probably this could be achieved by adding requestId to record['context'] through Monolog/Processor/UidProcessor. However this particular Processor forces us to provide integer ids. https://github.com/Seldaek/monolog/blob/master/src/Monolog/Processor/UidProcessor.php We'd have two alternatives: 1. Implement our own Monolog/Processor/RequestIdProcessor based on UidProcessor.php or 2. Alter requestId in Core\Bootstrap to integer I'd opt for 1.
  13. Susi Moog commented on patch set 5 05/17/16, 18:20

    For the request ID part: Would you say either https://github.com/Seldaek/monolog/blob/master/src/Monolog/Processor/ProcessIdProcessor.php or https://github.com/Seldaek/monolog/blob/master/src/Monolog/Processor/WebProcessor.php would provide enough information? if not I'd also add our own processor.

    DefaultConfiguration.php:1150 Reply

    We'd have to either add it directly here (which "feels" ugly) or we'd have to write our own stream handler as far as I can see. Will think on that a bit.
  14. Georg Ringer commented on patch set 4 05/17/16, 18:46

    Feature-12345-ReplaceLoggingWithMonolog.rst:49 Reply

    additionally, because there you can also state which processors and stuff is all removed and maybe replaced with new ones
  15. Helmut Hummel commented on patch set 5 05/17/16, 19:18

    LogManager.php:8

    I assume this is not the correct import
  16. Steffen Müller commented on patch set 5 05/17/16, 21:20

    The TYPO3 requestId is a random string which is unique for each request. It is created very early in TYPO3 bootstrap. It is useful to gather all log message which are logged in the same request. Neither Monolog ProcessIdProcessor nor WebProcessor work as substitute for the current requestId implementation in LogRecord. ProcessIdProcessor uses the id from the php process which is not unique per request. WebProcessor requires the ID to be set in $_SERVER['UNIQUE_ID'], but TYPO3 should set the requestId for convenience reasons. Most users do not have the possibility to add custom fields to $_SERVER
  17. Steffen Müller commented on patch set 5 05/17/16, 21:24

  18. Susi Moog commented on patch set 5 05/18/16, 09:25

    Thanks Steffen, for the processor thing. I will continue to accumulate your comments during the week and try to come up with a patch that addresses them during the weekend.
  19. Susi Moog abandoned the change 08/02/16, 10:08

  20. Susi Moog restored the change 07/06/18, 14:27

  21. Alexander Schnitzler uploaded patch set 6 07/07/18, 10:32

  22. TYPO3com voted on patch set 6 V -1 07/07/18, 10:32 Completed build in 06s on Sat, 7 Jul, 12:32 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-25208

  23. Alexander Schnitzler uploaded patch set 7 07/07/18, 10:37

  24. TYPO3com voted on patch set 7 V -1 07/07/18, 10:37 Completed build in 07s on Sat, 7 Jul, 12:37 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-25209

  1. Alexander Schnitzler uploaded patch set 8 07/07/18, 10:42

  2. TYPO3com voted on patch set 8 V -1 07/07/18, 10:50 Completed build in 07m 57s on Sat, 7 Jul, 12:50 PM Test Summary: 63 of 4433 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-25210

  3. Alexander Schnitzler commented on patch set 8 07/07/18, 10:51

    Alright. Bamboo failed as I didn't fix the tests yet but I guess the latest patch set is stable enough to discuss the implementation. I had a look at monolog and I am familiar with the configuration of monolog in symfony and that's what influenced this patch set a lot. The api for the user doesn't change at all but the way we look at the configuration differs a lot from the former approach. monolog is mainly about channels. Every logger is assigned to just one channel and the handlers are configured to listen to the channels. The former approach was a bit similar except that every class represented a channel which is not really helpful. Please review concept wise and let me know if you like the approach and if we should put more effort into this. I guess it makes sense to focus on the changes in the LogManager and DefaultConfiguration at first.
  4. Alexander Schnitzler uploaded patch set 9 07/07/18, 19:19

  5. TYPO3com voted on patch set 9 V -1 07/07/18, 19:27 Completed build in 07m 33s on Sat, 7 Jul, 09:27 PM Test Summary: 6 of 32975 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-25220

  6. Benni Mack commented on patch set 9 07/09/18, 10:47

    LocalImageProcessor.php:28

    we could use a trait here directly (separate patch)

    FileProcessingService.php:47

    we could use a trait directly here

    StorageRepository.php:58

    same here!
  7. Susi Moog rebased onto patch set 10 07/09/18, 11:27

  8. TYPO3com voted on patch set 10 V +1 07/09/18, 11:36 Completed build in 08m 48s on Mon, 9 Jul, 01:36 PM Test Summary: 32727 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-25248

  9. Markus Klein commented on patch set 10 07/09/18, 12:56

    _<hash> missing
    not
    available
    this should be part of the patch already if we replace our current framework. the current processors need an equivalent too. (WebProcessor)
    we have that now, IMO it has to be there in the new framework from the very beginning
    improves
  10. Alexander Schnitzler commented on patch set 10 07/09/18, 14:22

    Reply

    To be honest here. I dislike the security by obscurity idea. I'd rather add a rule to the example .htaccess, disallowing access to all .log files.

    Reply

    Yes, you are right. I'd just not put work into it right now until we have an agreement about using monolog or not. This patch set is just a proof of concept at some points.

    Reply

    Same answer like some lines above. Once we decided for monolog, we tackle these things as well.
  11. Helmut Hummel commented on patch set 10 07/09/18, 14:24

    DefaultConfiguration.php:1308

    if we put instances directly in configuration, we remove the possibility to have other configuration formats than PHP. besides that, the configuration should imho reflect a feature set we need (channels, log levels), but abstract from a concrete implementation. Otherwise we bind TYPO3 and the configuration to a concrete implementation (in this case a concrete version of monolog), which completely obsoletes the idea/possiblity of depending on an abstraction (psr-3) If we still decide to go this path, then the logging configuration must at least be removed from TYPO3_CONF_VARS (main TYPO3 configuration) and put elsewhere
  12. Alexander Schnitzler commented on patch set 10 07/09/18, 14:41

    DefaultConfiguration.php:1308 Reply

    > if we put instances directly in configuration, we remove the > possibility to have other configuration formats than PHP. True. The idea behind this configuration is, that we do bind to concrete instances anyway, no matter if we declare a classname or an instance. The good thing about using instances here (for now), is that the user has a clear api through autocompletion of his IDE. > besides that, the configuration should imho reflect a feature set > we need (channels, log levels), but abstract from a concrete > implementation. Please explain that again. I don't get it. > Otherwise we bind TYPO3 and the configuration to a concrete > implementation (in this case a concrete version of monolog), which > completely obsoletes the idea/possiblity of depending on an > abstraction (psr-3) Well, we do bind to any kind of monolog api no matter if we directly use concrete instances or use some kind of configuration in between. In the end we rely on monolog following semantic versioning and not changing the api. The statement about psr-3 is wrong because that psr does not describe how to deal with the configuration. Neither does it deal with handlers, processors and such. But true, if we want to use monolog, we rely on it and its api regarding the configuration. > If we still decide to go this path, then the logging configuration > must at least be removed from TYPO3_CONF_VARS (main TYPO3 > configuration) and put elsewhere I don't get that as well. -- Some general statements on configuration. For years we didn't have a plan on how to configure at all. Now, there is a movement towards yaml but I don't know the current status. While I worked on this patchset I had a look at symfony and I'd be glad to be able to configure like symfony does. Unfortunately it's not possible. If we had a clear statement about that topic I would have followed a different approach. For now I think, it's way easier using concrete instances in the configuration than having a mapping in between like 'type' => 'stream', which then has to be mapped and you write a huge case statement, mapping keys to class names. I simply don't see the benefit right now though I am open for better ideas.
  13. Markus Klein commented on patch set 10 07/09/18, 16:34

    > if we put instances directly in configuration, we remove the possibility to have other configuration formats than PHP. The point is that some of use (eg Helmut) are using different configuration mechanisms, which "generate" the PHP configuration. If we put PHP objects in there, this is getting a bit harder. I guess that's what Helmut was referring to.
  14. Benni Mack uploaded patch set 11 07/11/18, 09:38

  15. Benni Mack commented on patch set 11 07/11/18, 09:39

    simply rebased so the impact is better to understand
  16. TYPO3com voted on patch set 11 V +1 07/11/18, 09:45 Completed build in 07m 01s on Wed, 11 Jul, 11:45 AM Test Summary: 32727 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-25288

  17. Helmut Hummel wrote 07/15/18, 09:50

    Removed reviewer Helmut Hummel.
  18. Christian Kuhn abandoned the change 09/16/18, 22:53

    i'm sad to say this patch is abandoned the second time. it still seems no final conclusion has been reached and no solution has been established for v9. it would be great to see this patch resurrected for v10 again, for now i'll abandon it for the sake of a more clean review queue.
  19. Markus Klein commented on patch set 11 09/17/18, 14:00

    It is for sure not dead. A lot has been achieved concept-wise. So, this will get alive again!
  20. Arno Schoon commented on patch set 11 03/19/19, 20:47

    I think I agree with Helmut regarding the configuration, sure it feels funny to add executable code to a configuration file, but then again, that's the beauty of PHP. But more important there is this benefit of some auto-completion and giving people a clear grasp of what's going on. I'm not sure what should be done with the two TODO items mentioned, documentation and a config migration wizard. In my opinion it suffices to mark this as breaking because of a conceptual change, but for other developers pointing to things like https://github.com/Seldaek/monolog/blob/master/doc/01-usage.md could be more helpful than writing it ourselves. If you mean examples and specific use cases I'm in, but that doesn't need to hold this patch back as long as the implementation and parts of the Core are a good example. I think the changelog is clear about the rationale of this implementation. As for a migration wizard, is it really just a matter of changing the configuration or making developers aware of the new concept by detecting legacy configuration and giving (static) hints on how to improve it (maybe with a few use cases). Please let me know your thoughts on this.
  21. Benni Mack commented on patch set 11 03/20/19, 05:34

    Hey Arno, thanks for the reply. I consider the patch actually too large. First, I suggest that we move our Core APIs (LogLevel) to be fully PSR-3 compliant, which we aren't at all. Once this is done, we con continue with further steps. There is also a channel #cig-monolog in slack. Feel free to contact Markus, as he's done some more work in this area in the past weeks.
  22. Arno Schoon commented on patch set 11 03/20/19, 09:19

    > Patch Set 11: > > Hey Arno, > > thanks for the reply. I consider the patch actually too large. First, I suggest that we move our Core APIs (LogLevel) to be fully PSR-3 compliant, which we aren't at all. > > Once this is done, we con continue with further steps. > > There is also a channel #cig-monolog in slack. Feel free to contact Markus, as he's done some more work in this area in the past weeks. Agree, let's migrate to PSR-3 to clean up the code and remove some "proprietary" interfaces. This conceptual change will remain just as big I guess. I understood from Markus that he already was working on such a PSR-3-patch, lets see If we can get this patch smaller anyway :) As for configuration, this occurred too me this late last night, this patch of `DefaultConfiguration.php` doesn't allow me (as a system maintainer or administrator) to modify the path where the logs are stored. At least `LocalConfiguration.php` doesn't. I could choose to mis-use `AdditionalConfiguration.php` or even an extension, but that's rather late in the bootstrapping process which might lead to orphaned instances or logs written to the wrong location early in the process.
  23. Alexander Schnitzler restored the change 11/09/19, 13:52

  24. Susi Moog marked it work in progress 02/28/20, 07:37

  25. Susi Moog abandoned the change 02/28/20, 09:47

    abandoned for now in favor of the approach in https://review.typo3.org/c/Packages/TYPO3.CMS/+/62355