Review main Merged

72048[BUGFIX] Fix too long regular expression error in ContentObjectRenderer

Anja Leichsenring CR +2 V +2 Normal +60 −19 updated 11/10/21, 18:57

Commit message

Text cropping on texts with way more than 1000 characters
triggers regular expression compilation errors.
The sweet spot was determined to a value of max 962 characters,
thus leading to internal 'preg_match' error state and throwing
'Compilation failed: regular expression is too large' warning.

The patch crops text in multiple chunks and reassembles the
chunks until crop position was reached to avoid this internal
error exceeding the max regular expression pattern length.

Further tests are added to cover this scenario.

Review

  • Code review CR +2 Done
    • Christian Kuhn +2
    • Nikita Hovratov +1
  • Verified V +2 Done
    • Christian Kuhn +2
    • Nikita Hovratov +1
    • core-ci +1 CI

Issues 1

Files 2

Review log 3341

Show 1216 earlier messages
  1. Anja Leichsenring uploaded patch set 1 11/04/21, 11:24

  2. core-ci voted on patch set 1 V -1 11/04/21, 11:46 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18867

  3. Stefan Bürk uploaded patch set 2 11/04/21, 12:53

  4. Stefan Bürk commented on patch set 2 CR -1 11/04/21, 12:55

    One first solution - but not sure, if it is really good to do it that way to prevent the compilation error - or if there are untested/unregonized sideeffects. Maybe someone can take this as a startpoint - can look into it further after work.
  5. Stefan Bürk wrote 11/04/21, 12:56

    Removed Code-Review-1 by Stefan Bürk <stefan@buerk.tech>
  6. Stefan Bürk voted on patch set 2 CR -1 11/04/21, 12:56

  7. core-ci voted on patch set 2 V +1 11/04/21, 13:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18878

  8. Simon Gilli uploaded patch set 3 11/04/21, 15:00

  9. Anja Leichsenring commented on patch set 3 11/04/21, 15:07

    ContentObjectRenderer.php:3046

    why 1000? The value has been randomly chosen by me to be 'above breaking point'. The highest value no error is raised is 962.

    ContentObjectRenderer.php:3050

    while you are on it: queal should be equal ;)
  10. Stefan Bürk commented on patch set 3 11/04/21, 15:12

    ContentObjectRenderer.php:3046 Reply

    In the issue was something with little over 1000. As it was a first try in ~15mins in the launch break. So not take this as the real number. Just saving a first try. Simon has some more insights, and talking with him - he has a play. But one point would be to determine on which value this must be done etc.

    ContentObjectRenderer.php:3050 Reply

    Ack. Sure ;)
  11. core-ci voted on patch set 3 V -1 11/04/21, 15:20 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18896

  12. Simon Gilli commented on patch set 3 CR -1 11/04/21, 15:21

    Stefan and I are on it. So fare I have fixed the tests to show the issue with the current solution. We will hopefully come up with the right solution this evening... The rest is discussed at Slack for the time beeing. And another side note, the best solution here would probably be using a DOM parser and not abuse regex for it. But that's another story so for the moment it's fine to just fix the current code.
  13. Jonas Eberle commented on patch set 3 11/05/21, 12:44

    Reply

    Maybe \TYPO3\CMS\Core\Html\HtmlParser can help here to simplify the logic in that function?
  14. Stefan Bürk uploaded patch set 4 11/05/21, 15:40

  15. core-ci voted on patch set 4 V +1 11/05/21, 15:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18960

  16. Stefan Bürk uploaded patch set 5 11/05/21, 17:59

  1. Stefan Bürk commented on patch set 5 11/05/21, 18:02

    Added fix on the right place now - took little longer, as it was a little tricky.

    ContentObjectRenderer.php:3046 Reply

    Done

    ContentObjectRenderer.php:3050 Reply

    Done
  2. Stefan Bürk wrote 11/05/21, 18:04

    Removed reviewer Stefan Bürk.
  3. core-ci voted on patch set 5 V +1 11/05/21, 18:20 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/18966

  4. Simon Gilli voted on patch set 5 V +1 CR +1 11/06/21, 10:04

  5. Christian Kuhn updated the commit message in patch set 6 11/09/21, 17:18

  6. Christian Kuhn voted on patch set 6 V +1 CR +1 11/09/21, 17:18

  7. core-ci voted on patch set 6 V +1 11/09/21, 17:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19078

  8. Nikita Hovratov commented on patch set 6 11/10/21, 00:59

    ContentObjectRenderer.php:3072

    This is the culprit btw (the pipe). This causes some sort of backtracking states (not an regex expert, just read some stackoverflow: https://stackoverflow.com/questions/33976303/php-why-am-i-being-warned-that-my-regular-expression-is-too-large/33977966#33977966). I guess this causes us to reach this 64k limit fairly easy. Probably also related to this 962 value. Best solution would be to get rid of it somehow. Have played around a bit, but it's not so easy to satisfy all test cases with a new implementation.
  9. Nikita Hovratov commented on patch set 6 11/10/21, 03:18

    ContentObjectRenderer.php:3071

    Should we use mb_substr here?
  10. Stefan Bürk commented on patch set 6 11/10/21, 07:27

    ContentObjectRenderer.php:3071 Reply

    Sure, thanks for pointing out. Will change that.

    ContentObjectRenderer.php:3072 Reply

    Played arround with regex and other options too for some hours. The first group is there to find the htmlentitie styles and count them as 1 instead of the full length, which compiles in a broader way. And in combanitation with the "u" modifier flag for unicode this get's even worser. This whole method should get a refactor as already mentioned, as doing these regexp things on html is always nasty. I do not see to get a regexp here which do which has do be done and don't reach the limit. Normal way would be to splitt each group part out and do it on the own, which would make it even worser. I would say, that the current solution with splitting the to chunks is a solid working solution for now (as bugfix) - and gives us time to implement a working refactor/reimplemantion.
  11. Stefan Bürk uploaded patch set 7 11/10/21, 07:43

  12. Stefan Bürk commented on patch set 7 11/10/21, 07:46

    ContentObjectRenderer.php:3071 Reply

    Done
  13. core-ci voted on patch set 7 V +1 11/10/21, 07:55 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19102

  14. Nikita Hovratov commented on patch set 7 11/10/21, 09:21

    ContentObjectRenderer.php:3072 Reply

    Yeah, this is what I understood too. This can't be fixed with another regex I think. The chunk solution is a valid bugfix, thanks for that! I maybe have a clean(er) solution, at least all test cases are green now. The only regex is now to find html entities. Not sure about unicode though. Neverheless, let's get this bugfix in first as it's also for v10. And then we can see, if this can be improved on master.
  15. Stefan Bürk commented on patch set 7 11/10/21, 10:09

    ContentObjectRenderer.php:3072 Reply

    I think having that chunk solution as working (first) would be a good thing, as you said. Anyway I'm interesseted in your solution .. may you share your "regexp" here as comment - or maybe ping me on slack ?
  16. Nikita Hovratov voted on patch set 7 V +1 CR +1 11/10/21, 10:12

  17. Stefan Bürk commented on patch set 7 11/10/21, 11:06

    ContentObjectRenderer.php:3072 Reply

    ^ done. We take this as safe solution.
  18. Stefan Bürk commented on patch set 7 11/10/21, 11:07

    ContentObjectRenderer.php:3072 Reply

    done
  19. Sybille Peters commented on patch set 7 11/10/21, 12:20

    Just a minor comment: It sounds like we are fixing preg_match here ;-) Maybe "Fix error using preg_match" or something.
  20. Stefan Bürk updated the commit message in patch set 8 11/10/21, 12:24

  21. Stefan Bürk commented on patch set 8 11/10/21, 12:25

    Reply

    Done
  22. core-ci voted on patch set 8 V +1 11/10/21, 12:36 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19123

  23. Nikita Hovratov voted on patch set 8 V +1 CR +1 11/10/21, 13:36

  24. Christian Kuhn voted on patch set 8 V +2 CR +2 11/10/21, 18:57

  25. Christian Kuhn merged patch set 9 11/10/21, 18:57