Review main Abandoned

76003[BUGFIX] Catch exception to prevent flooding the log file

Frank Nägler CR 0 V 0 Easy +28 −1 updated 05/10/23, 17:11

Commit message

The Symfony expression language parser does not take care of
non-exisiting array keys. For that reason, the TYPO3 log is flooded
with "array key does not exist" messages.
The exception is caught and checked to prevent this, only for this error.

This fix is an ugly workaround that solves the main issue and can
be removed after Symfony adds checks to the parser.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 2

Review log 2024

  1. Frank Nägler uploaded patch set 1 10/07/22, 18:21

  2. Oliver Klee commented on patch set 1 10/07/22, 18:25

    Is there already a ticket (or even PR) for this problem on the Symfony side?
  3. core-ci voted on patch set 1 V +1 10/07/22, 18:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/32082

  4. Frank Nägler commented on patch set 1 10/07/22, 18:36

    Yes, there is an issue that describes the issue. The solution was merged int 6.2 branch as feature "Add support for null coalescing" https://github.com/symfony/symfony/issues/21691
  5. Frank Nägler commented on patch set 1 10/07/22, 18:38

    Ao, this could be a solution for 12 LTS, if we update to Symfony 6.2 (release in November 2022) but not for v11. So I think we should go with this solution.
  6. Frank Nägler uploaded patch set 2 10/07/22, 18:52

  7. Frank Nägler commented on patch set 2 10/07/22, 18:53

    I have added an additional comment to the workaround. If we update the Symfony components to 6.2 we can maybe remove this workaround. If we do the update to 6.2 before 12 LTS, this will only be kept for v11.
  8. core-ci voted on patch set 2 V -1 10/07/22, 19:00 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/32083

  9. Frank Nägler uploaded patch set 3 10/07/22, 19:11

  10. Frank Nägler commented on patch set 3 10/07/22, 19:13

    ResolverTest.php:105

    This test case fails with: Undefined array key "undefined" The code (fix) prevents exactly this message, but not in the tests, I have no idea how to solve this for the test. Maybe someone else has an idea?
  11. core-ci voted on patch set 3 V -1 10/07/22, 19:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/32084

  12. Oliver Klee wrote 10/07/22, 19:38

    Removed cc Oliver Klee.
  13. Stefan Bürk commented on patch set 3 10/08/22, 00:04

    I'm not a fan of this workaround introducing a exception wrapper and silencing the log entries. symfony/expression-language 6.2+ won't fix this issue magically and needs user work like now to fix the issue. Removing this exception after requiring symfony/expression-language 6.2 will return to spam the log with messages, if people do not use the ?? operator in there typoscript. Thus simply "moving" this issue forward. The "recommended" solution for now is, if these errors spam the log to use the traverse() workaround in typoscript. Both needs the the integrator to adjust(fix) the corresponding TypoScript. Using this "workaound" will silence this issue, so people "won't" fix there TyposScript until then. The documentation team already started to change the documentation and use travers for other TypoScript options to mitigate this, for example: https://docs.typo3.org/m/typo3/reference-typoscript/main/en-us/Conditions/Index.html#page https://docs.typo3.org/m/typo3/reference-typoscript/main/en-us/Conditions/Index.html#page With symfony/expression-language 6.2 there will be another ability to mitigate this. But people need the logs to know that they have to do something. This change will only transport the "issue" to the future. We already recommended a lot to use the traverse() workaround to fix the TypoScript in the slack channels. What do you think ?
  14. Stefan Bürk commented on patch set 3 10/08/22, 01:13

    ResolverTest.php:105 Reply Unresolved

    In tests, the undefined array key access warning/notice gets converted to an exception. In "Resolver" the try-catch catches `\TYPO3\CMS\Core\Error\Exception` which may be the wrong exception. This the tests still failes as it is not covered by the try-catch in Resolver->evaluate().
  15. Frank Nägler commented on patch set 3 10/08/22, 06:16

    Reply

    I know the traverse function, but as stated in the forge ticket, I think blowing up the conditions with additional function calls is not user-friendly. At the moment we have clean and valid conditions that worked in the past (and now) but now flooding the logs because of the array key thingy. Blowing up the conditions with the traverse function is for me a workaround too, and makes the conditions ugly and harder to read and understand. The patch saves hundreds or thousands of people's time and work, and makes the system easier to use. From my point of view: think less technical and more user-friendly. The technical impact on our side is so minimal, but the impact on our users is huge.
  16. Stefan Bürk commented on patch set 3 10/08/22, 10:52

    Reply

    It's the same as the "undefined array key" epic in code's. So I get you right - you won't remove the exception handling in the future and leave the people in the end with "unsafe" code ? And let it to the people hopfefully and blindly using proper checks (??/??= with expression language 6.2) ? From user/agency perspective - yeah it was anoying, but after tackling and fixing these issues in project it is clean. And safe for backend usage (where most of these errors occured anyway). This would not have been possible without the log entries. Maybe a swithc would be fine - so people can activate it (or supress it) but still have the ability to fix these places based on the log entries.
  17. Markus Klein commented on patch set 3 10/08/22, 19:41

    Reply

    As the original author of the traverse() function: If I remember correctly, I implemented this in order to have "some means" of avoiding chains like `[request.getQueryParams() && request.getQueryParams()['news'] && request.getQueryParams()['news']['newsId]']`. I understood, back then, that the expression language stuff was actually made for devs, having control over the PHP code and not integrators in the TYPO3 sense, using this is a totally different context/language/syntax, where "error handling" is expected to behave different aka less strict than in PHP. So yes, the traverse() function evolved as a necessity to have a fast fix for this problem and could be included in any project immediately. It finally found its way into the Core. I would be totally happy if we can get rid of my traverse() invention in the midterm. If the exception handling, as suggested here, does the trick for the time being that is fine with me. I really don't see the comparison/relation with the other "undefined array key" fixes for PHP code. This is integrator-land, which can be a bit relaxed than what I require from PHP devs.
  18. Frank Nägler commented on patch set 3 10/08/22, 20:09

    Reply

    Of course, the fix that catches the exception here can and should be removed again. If 6.2 is in the core, you can use the clean ?? syntax. And to be clear: conditions are user/integrator land. An integrator does not and should not be a PHP developer to know the depths of PHP to build a website with conditions. The Expression Language is clearly not a developer feature. I have nothing against the traverse function per se, but an integrator shouldn't have to understand what "array key not exist" messages are all about. A non-dev can't do anything with the message in the logs. I think a feature switch/toggle is ok, and as a result of the discussion, I'm currently considering whether a meaningful message in the log would make sense (if logging hasn't been deactivated). A message explains what the problem is and how to solve it.
  19. Stefan Bürk commented on patch set 3 10/08/22, 20:45

    Reply

    > Of course, the fix that catches the exception here can and should be removed again. If 6.2 is in the core, you can use the clean ?? syntax. And that's the point I don't really get. If we would remove the exception catching with expression-language 6.2 again, it will be the same situation as right now. The same "integrator does not understand" log entries are back and spaming again. What makes than the different here to use traverse() for now, and having with that 2 variants to make TypoScript safe than - and having the logs to state "there is something of" which has to be addressed ? Again, The same undefined arrey error logs would be back when removed .. so same situation. So why introducing it ? It just make the logs for some days silent and move it forward in time ending in the same situation like now. If you really want to safe-guard integrators the exception should be catched and "translated" ... from "undefine array key access" to something more hint-full for integrators, at best to some explanations with example what may be and how to solve it (know and than). That would be more useful than simply suppress it completly as it would leave the errors in thy TypoScript. Adding and removing it is not a solution, it's just a timewarp for this discussion and it will be back than. Just saying ...let them fix that earlier than later. Just my five cents. Don't get me wrong - I get the point for the integrators. But at some point the have to deal with it. The only difference is that I'm saying let them hurt now instead of then - let them learn now to do it right. Or leave the exception catching for ever. Personally not seeing the benefit of it. But let's close this. I stated my opinion on this. And it was not meant as a blocker.
  20. Frank Nägler commented on patch set 3 10/10/22, 11:55

    Reply

    Here are the main points from my point of view: 1. traverse is a workaround (or a helper, I don't know) 2. ?? is the defined solution from Symfony and will be released with 6.2 in November 2022 3. If we can or will switch from Symfony 6.1 to 6.2 is a future decision and maybe only possible with v13. That would mean the users have to live with that for years and not only for some days. 4. as mentioned, the exception must be transformed into a helpful message, like: "hey use traverse()" or "hey use ?? operator in the future". I don't know how to progress here because I am not a core merger; the core team has to make the final decision.
  21. Benni Mack updated the commit message in patch set 4 V -1 05/03/23, 09:24

  22. core-ci voted on patch set 4 V -1 05/03/23, 10:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41406

  23. Christian Kuhn commented on patch set 4 CR -1 05/10/23, 17:11

    I don't think we should fix this anymore, and i'll abandon the change: The decision is to let such errors bubble up. Integrators need to sanitize their conditions instead. Documentation in this area has been vastly improved as well. The solution is a hack and we now have two mitigations: either use traverse() as documented, or use the ?? operator that is available since symfony v6. core v11 still contains a hack to suppress this stuff btw, so only v12 is affected, and the behavior is documented. I hope it's ok to abandon here.
  24. Christian Kuhn abandoned the change 05/10/23, 17:11