Review main Merged

84909[BUGFIX] Allow using parentheses in TypoScript operator functions

Garvin Hicking CR +2 V +2 Hard +303 −9 updated 11/14/24, 12:32

Commit message

The TypoScript Parser was not able to parse this:

foo := appendString( AND `uid` NOT IN (48,10,50))

It recogized the first closing parentheses ")" as
function end. The patch fixes this by counting the
number of parentheses to avoid early stop. This
approach is limited, but it establishes backwards
compatibility for most use cases.

A better solution is to have "inner" parentheses
quoted to cope with cases where the number of opening
and closing parentheses don't match, but that's a syntax
addition that needs to be done with a separate patch.

Limits of current approach: In case of extra closing
parentheses, any string is ignored past the first valid
match:

foo = Do
foo := appendString( something(withExtra())))))))))))
// Result: "Do something(withExtra())"

In case of extra opening parantheses and no matching
closing parentheses, the result will be an
empty string/stream:

foo = Do
foo := appendString( something(((((((()
// Result: "Do"

Note multiline assignments remain invalid:

foo = Do
foo := appendString(
something
)

Review

  • Code review CR +2 Done
    • Christian Kuhn +2
    • Oli Bartsch +1
  • Verified V +2 Done
    • Christian Kuhn +2
    • Oli Bartsch +1
    • core-ci +1 CI

Issues 4

Files 4

Review log 2733

Show 68 earlier messages
  1. Garvin Hicking uploaded patch set 1 06/25/24, 13:39

  2. core-ci voted on patch set 1 V +1 06/25/24, 13:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/60304

  3. Garvin Hicking rebased onto patch set 2 V +1 07/28/24, 07:22

  4. Georg Ringer voted on patch set 2 CR +1 V +1 07/28/24, 07:22

  5. core-ci voted on patch set 2 V +1 07/28/24, 07:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/62344

  6. Christian Kuhn commented on patch set 2 CR -1 07/29/24, 13:01

    this needs tests in typo3/sysext/core/Tests/Unit/TypoScript/Tokenizer/TokenizerInterfaceTest.php to show/verify the token stream.
  7. Garvin Hicking uploaded patch set 3 CR +1 V +1 07/30/24, 17:07

    Add TokenizerInterfaceTest cases
  8. Garvin Hicking commented on patch set 3 07/30/24, 17:08

    Reply

    Added a few tests. Unsure about that one place I marked with "@todo", please dadvise. :-)
  1. Garvin Hicking commented on patch set 3 07/30/24, 17:08

    TokenizerInterfaceTest.php:3654

    ^^ here
  2. core-ci voted on patch set 3 V +1 07/30/24, 17:12 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/62462

  3. Georg Ringer voted on patch set 3 CR +1 V +1 07/31/24, 15:12

  4. Garvin Hicking commented on patch set 3 08/21/24, 19:39

    (Discussed with Lolli. Will have a look at this after Feature Freeze)
  5. Garvin Hicking rebased onto patch set 4 CR +1 V +1 09/18/24, 09:19

  6. core-ci voted on patch set 4 V +1 09/18/24, 09:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/66112

  7. Benni Mack updated the commit message in patch set 5 V +1 10/18/24, 13:53

  8. Christian Kuhn commented on patch set 5 10/21/24, 09:52

    missed this one when i looked at open ts parser issues last week. oli pointed me to it, we should get it done. will have a closer look soon.
  9. Garvin Hicking commented on patch set 5 10/21/24, 13:00

    Reply

    Thanks. I'm a bit afraid of this "counting parantheses" approach, even though test says it seems to work. So I'm open to feedback. We should check if a bugfix is something to backport to v12 (I'd say yes).
  10. Oli Bartsch commented on patch set 5 10/21/24, 14:20

    Reply

    This should definitely be backported to v12.
  11. Christian Kuhn commented on patch set 5 11/11/24, 14:56

    Reply

    i think we should add tests to see what happens if there is a comment that contains "(" and/or ")" after a := function.
  12. Christian Kuhn commented on patch set 5 11/11/24, 14:57

    i'll try to work on this tomorrow.
  13. Christian Kuhn commented on patch set 5 11/13/24, 15:53

    TokenizerInterfaceTest.php:3654 Reply

    That strategy is done at multiple places in the parser: when it knows a line is "finished" at some point, and there is some left-over, it is treated as comment. I guess its fine to do this here as well.
  14. Christian Kuhn commented on patch set 5 11/13/24, 16:23

    Hey. Yes, this fixes the issue for situations like this: foo := addToList(a, (, )) and foo := appendString( AND `uid` NOT IN (48,10,50)) I think we should do this since it re-establishes b/w compat - it worked before like that. The solution is based on the fact that there is the name number of opening and closing parenthesis within the () encapsulated function body. However, this breaks: foo := addToList(a, (, b) and foo := addToList(a, ), b) Stuff like this can only be solved in a clean way, when the special character within the body are quoted. We have some quote logic in the parser with identifiers already: foo.bar\.baz is segment 'foo' and segment 'bar.baz'. We could eventually add this with another patch: '\(' ignores the '(' and treats it as literal, as well as '\)', '\\(' would be '\(', and '\\)' would be '\)'. We could state that quoting '(' and ')' is best practice and should always be done to avoid issues. What do you think?
  15. Christian Kuhn commented on patch set 5 11/13/24, 16:24

    Reply

    args. gerrit ate the backslashes away.
  16. Christian Kuhn commented on patch set 5 11/13/24, 16:31

    Reply

    an identifier "foo.bar\\.baz" is segment "foo" and segment "bar.baz". proposal: "\\(" takes the special meaning of "(" away in function body, as well as "\\)" for ")". This would be an addition to this patch, and a better best practice we could communicate. And since constant substitution happens after tokenizing, it shouldn't be an issue when a constant contains parenthesis.
  17. Christian Kuhn uploaded patch set 6 11/14/24, 11:45

  18. Christian Kuhn commented on patch set 6 11/14/24, 11:45

    Reply

    Done

    Reply

    Done

    Reply

    Acknowledged

    Reply

    there is a simple test for "left over stuff is turned into a comment". that should be fine for now.
  19. Christian Kuhn updated the commit message in patch set 7 11/14/24, 11:46

  20. core-ci voted on patch set 6 V +1 11/14/24, 11:52 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69429

  21. core-ci voted on patch set 7 V +1 11/14/24, 11:53 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/69430

  22. Christian Kuhn voted on patch set 7 CR +1 V +1 11/14/24, 12:21

  23. Oli Bartsch voted on patch set 7 CR +1 V +1 11/14/24, 12:26

  24. Christian Kuhn voted on patch set 7 CR +2 V +2 11/14/24, 12:32

  25. Christian Kuhn merged patch set 8 11/14/24, 12:32