Review main Merged

82633[BUGFIX] Add constant evaluation in TypoScript functions

Garvin Hicking CR +2 V +2 Hard +386 −78 updated 06/05/24, 10:07

Commit message

We missed a TypoScript feature with the new parser.
Function operator modifier values can use constants:

foo = 42
foo := addToList({$my.constant})

The solution is two fold: Until now, the value of a
function body was a single T_VALUE token. This is
turned into a stream of tokens that can consist of
multiple T_VALUE and T_CONSTANT tokens. This is
handled in the tokenizers. Secondly, the AST builders
need to deal with these token streams now, and take
care available constants are hand over to
ConstantAwareTokenStream before the stream is cast
to string and single functions are evaluated.

The BE modules *do* reflect constant substitutions in
function modifier bodies, but not in an ideal way.
An @todo within the responsible CommentAwareAstBuilder
outlines ideas how this could be improved.

Note TYPO3 v13 supports constant fallbacks here
as well, this syntax feature came with #103671.
This works, but is NOT supported in v12:

foo := addToList({$my.constant ?? $fallback.constant})

Review

  • Code review CR +2 Done
    • Christian Kuhn +2
    • Garvin Hicking +1
    • Stefan Bürk +1
  • Verified V +2 Done
    • Christian Kuhn +2
    • Garvin Hicking +1
    • Stefan Bürk +1
    • core-ci +1 CI

Issues 3

Files 11

Review log 6095

Show 4370 earlier messages
  1. Garvin Hicking uploaded patch set 1 01/24/24, 12:45

  2. Garvin Hicking marked it work in progress 01/24/24, 12:46

  3. core-ci voted on patch set 1 V -1 01/24/24, 12:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52078

  4. Garvin Hicking uploaded patch set 2 V -1 01/24/24, 15:25

    Implement CGL
  5. core-ci voted on patch set 2 V -1 01/24/24, 15:33 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52092

  6. Garvin Hicking uploaded patch set 3 V -1 01/24/24, 16:06

    Address bad node getValue
  7. core-ci voted on patch set 3 V -1 01/24/24, 16:12 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52097

  8. Garvin Hicking uploaded patch set 4 V -1 01/24/24, 20:46

    Adress test errors
  9. core-ci voted on patch set 4 V -1 01/24/24, 20:51 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52110

  10. Garvin Hicking uploaded patch set 5 V -1 01/24/24, 20:54

    Adress duplicate test names
  11. core-ci voted on patch set 5 V +1 01/24/24, 21:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52111

  12. Garvin Hicking uploaded patch set 6 V +1 01/25/24, 09:36

    Rework central parts add tests
  13. core-ci voted on patch set 6 V +1 01/25/24, 09:41 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52143

  14. Garvin Hicking updated the commit message in patch set 7 V +1 01/25/24, 09:45

    Adjust commit message
  15. Garvin Hicking marked it ready for review 01/25/24, 09:45

  16. core-ci voted on patch set 7 V +1 01/25/24, 09:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52144

  17. Garvin Hicking uploaded patch set 8 V +1 01/26/24, 06:22

    Bind to proper ticket
  18. core-ci voted on patch set 8 V +1 01/26/24, 06:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52211

  19. Christian Kuhn commented on patch set 8 02/09/24, 11:21

    LossyTokenizer.php:33

    please use TokenStreamInterface for type hints everywhere, that's what it is for. the single one place where you need to have that distinction to the comment-aware implemenation in the ast builder, we use the instanceof check and then we know.

    LossyTokenizer.php:590

    no we don't. addToList({$my,constant},42) will use the constant {$my,constant}. i don't see the point why you start looking at function value separators. the function tokenizer must not know about implementation details of single function value separators.

    LossyTokenizer.php:635

    please either keep the signature as is and always work on $this properties, OR change the tokenizer to always receive all arguments and never work on $this properties (this could be hard, there was a reason I went with the properties). But please to NOT implement a mixture of both.

    TokenizerInterfaceTest.php:3175

    this is broken. 17 is the correct value.

    TokenizerInterfaceTest.php:3183

    17
  20. Garvin Hicking uploaded patch set 9 V +1 02/12/24, 14:56

  21. core-ci voted on patch set 9 V -1 02/12/24, 15:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53086

  22. Garvin Hicking uploaded patch set 10 V -1 02/12/24, 15:21

  23. core-ci voted on patch set 10 V -1 02/12/24, 15:26 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53088

  24. Garvin Hicking uploaded patch set 11 V -1 02/12/24, 15:27

  25. core-ci voted on patch set 11 V -1 02/12/24, 15:36 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53093

  26. Garvin Hicking uploaded patch set 12 V -1 02/12/24, 15:42

  27. core-ci voted on patch set 12 V +1 02/12/24, 15:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53095

  28. Garvin Hicking commented on patch set 12 02/12/24, 16:16

    LossyTokenizer.php:33 Reply

    Ack. Totally went off-rails there, don't know why I didn't find this in first place.

    LossyTokenizer.php:590 Reply

    I see; I made it more complicated because I figured we should tokenize each function argument here too to access them one-by-one. I've now reworked it to just treat everything between "(" and ")" as a single function value now. (Note that something like `prependString(()` probably won't work (and doesn't now) with the simple parsing approach, if you wanted to use the closing parantheses character at any point within the function argument list. And we don't have string enclosure characters like " or ' in the TypoScript "specs" here to prevent it)

    LossyTokenizer.php:635 Reply

    This is the main pain point here. I've now reworked it to take mandatory method arguments now that indicate the "working scope" of the method. My current approach starts another TokenStream here, so that it can use the "bait and switch" method of how parseValueForConstances replaces a stream. Since I cannot replace the WHOLE stream, but just the function argument stream part of it, I needed a method to only work on that subpart. The subpart can't be another instance of Lossy/lessTokenizer to allow operating on `$this` because that would break the whole idea of a single Tokenizer service. To prevent needing a whole rework of more "DTOs" for a stream and refactoring the whole method, I tried to go a way with "least breakage" here. This is, BTW, the reason why on the places below, the context shifts from column "17" to a relative "7" one. When parsing "addToList(XXX)", it starts at character position 10 to parse XXX. The new substream starts with a 0-index (because "currentLineString" will be set to the function argument list, starting at 0). I do think we might want to look into this together some more, maybe you have another approach on how to solve the "branching off" of the function parsing here, so that a distinct stream can come out of it which binds the constants to that stream in the AST later on.
  29. Christian Kuhn commented on patch set 12 02/13/24, 08:59

    LossyTokenizer.php:590 Reply

    good point. we may need to allow quoting `)` as `\)` when that char is used as function value. The 'identifier' parsing does a similar thing for `.` (most commonly used in flex form details). should of course be a separate patch.

    LossyTokenizer.php:635 Reply

    yeah. i'm aware that handing over the subject data as argument destroys the character position of subsequent tokens. that's why i went with $this state in this method in the first place. i don't like it either. i guess we have to count the chars of the result or something to manipulate the char-counting for the tokens after the value stream.
  30. Garvin Hicking uploaded patch set 13 V +1 02/26/24, 08:52

    Adjust phpunit attributes resolve merge conflict
  31. core-ci voted on patch set 13 V -1 02/26/24, 08:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53587

  32. Garvin Hicking uploaded patch set 14 V -1 03/05/24, 10:32

  33. core-ci voted on patch set 14 V +1 03/05/24, 10:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53929

  34. Simon Schaufelberger commented on patch set 14 03/05/24, 10:54

    #102913 is not accessible. Why?
  35. Garvin Hicking commented on patch set 14 03/05/24, 18:53

    Reply

    Just to reduce mailspam. This is WIP, the ticket was just a stub. I did not want to flood the actual ticket # for this with WIP gerrit pushes - but now I see, the "related" line does trigger this. So this was a moot issue. I've unprivated the issue. ;)
  36. Simon Schaufelberger wrote 03/05/24, 19:10

    Removed cc Simon Schaufelberger.
  37. Garvin Hicking uploaded patch set 15 V +1 03/06/24, 08:17

  38. core-ci voted on patch set 15 V -1 03/06/24, 08:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53998

  39. Garvin Hicking uploaded patch set 16 V -1 03/06/24, 16:14

    Fix all the things
  40. Garvin Hicking commented on patch set 16 03/06/24, 16:16

    LossyTokenizer.php:635 Reply

    I have added an offset to this: ``` $this->parseValueForConstants($functionBodyPart, $functionValueStream, $functionBodyStartPosition); ... $valueToken = new Token(TokenType::T_VALUE, $currentString, $this->currentLineNumber, $this->currentColumnInLine + $lastTokenEndPosition + $tokenOffsetPosition); ```

    TokenizerInterfaceTest.php:3175 Reply

    Yes, reverted this line change and now uses offsets.

    TokenizerInterfaceTest.php:3183 Reply

    Done
  41. core-ci voted on patch set 16 V -1 03/06/24, 16:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54050

  42. Garvin Hicking uploaded patch set 17 V -1 03/06/24, 16:39

    Fix CGL
  43. core-ci voted on patch set 17 V +1 03/06/24, 16:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54056

  44. Marvin Buchmann voted on patch set 17 V +1 03/08/24, 09:24

  45. Chris Müller uploaded patch set 18 V +1 05/02/24, 17:55

  46. core-ci voted on patch set 18 V +1 05/02/24, 18:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57111

  47. Chris Müller rebased onto patch set 19 V +1 05/28/24, 11:04

  48. core-ci voted on patch set 19 V +1 05/28/24, 11:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58393

  49. Christian Kuhn uploaded patch set 20 V +1 05/28/24, 11:32

  50. Christian Kuhn commented on patch set 20 05/28/24, 11:33

    cosmetics and un-removing 2 test files that should stay.
  51. Christian Kuhn commented on patch set 20 05/28/24, 11:36

    TokenizerInterfaceTest.php:3573

    i think we should add a couple of tests to verify/show that functions with constants create proper token streams.
  52. core-ci voted on patch set 20 V +1 05/28/24, 11:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58394

  53. Christian Kuhn uploaded patch set 21 V +1 05/29/24, 08:55

  54. core-ci voted on patch set 21 V +1 05/29/24, 09:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58440

  55. Christian Kuhn uploaded patch set 22 V +1 06/04/24, 08:21

  56. Christian Kuhn commented on patch set 22 06/04/24, 08:22

    TokenizerInterfaceTest.php:3573 Reply

    Done
  57. core-ci voted on patch set 22 V +1 06/04/24, 08:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58667

  58. Christian Kuhn commented on patch set 22 06/04/24, 08:44

    LosslessTokenizer.php:859

    i wonder if we should just always set the stream and except in the line getter if it's null. this could simplify the ast builders a bit.
  59. Christian Kuhn uploaded patch set 23 V +1 06/04/24, 09:54

  60. core-ci voted on patch set 23 V -1 06/04/24, 10:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58674

  61. Christian Kuhn uploaded patch set 24 V -1 06/04/24, 10:31

  62. Christian Kuhn commented on patch set 23 06/04/24, 10:31

    LosslessTokenizer.php:859 Reply

    Done
  63. core-ci voted on patch set 24 V +1 06/04/24, 10:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58676

  64. Christian Kuhn uploaded patch set 25 V +1 06/04/24, 12:38

  65. Christian Kuhn uploaded patch set 26 06/04/24, 12:41

  66. core-ci voted on patch set 25 V +1 06/04/24, 12:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58679

  67. core-ci voted on patch set 26 V +1 06/04/24, 12:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58680

  68. Christian Kuhn uploaded patch set 27 V +1 06/04/24, 13:44

  69. core-ci voted on patch set 27 V +1 06/04/24, 13:49 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58681

  70. Christian Kuhn uploaded patch set 28 V +1 06/04/24, 16:25

  1. Christian Kuhn commented on patch set 28 06/04/24, 16:26

  2. core-ci voted on patch set 28 V +1 06/04/24, 16:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58688

  3. Christian Kuhn uploaded patch set 29 V +1 06/04/24, 18:31

  4. Christian Kuhn commented on patch set 29 06/04/24, 18:31

  5. core-ci voted on patch set 29 V +1 06/04/24, 18:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58699

  6. Christian Kuhn commented on patch set 29 CR -1 06/04/24, 18:39

    AstBuilderInterfaceTest.php:2815

    should assert ast, not its array representation.
  7. Christian Kuhn uploaded patch set 30 CR -1 V +1 06/05/24, 07:20

  8. Christian Kuhn commented on patch set 30 06/05/24, 07:21

    AstBuilderInterfaceTest.php:2815 Reply

    Done
  9. core-ci voted on patch set 30 V +1 06/05/24, 07:24 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58722

  10. Christian Kuhn uploaded patch set 31 V +1 06/05/24, 07:33

  11. core-ci voted on patch set 31 V +1 06/05/24, 07:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58724

  12. Christian Kuhn uploaded patch set 32 V +1 06/05/24, 07:41

  13. Christian Kuhn uploaded patch set 33 06/05/24, 07:43

  14. core-ci voted on patch set 32 V +1 06/05/24, 07:47 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58725

  15. core-ci voted on patch set 33 V +1 06/05/24, 07:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58727

  16. Christian Kuhn updated the commit message in patch set 34 V +1 06/05/24, 07:51

  17. core-ci voted on patch set 34 V +1 06/05/24, 07:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58728

  18. Christian Kuhn commented on patch set 34 CR -1 06/05/24, 09:21

    looking at 2 minor details.
  19. Christian Kuhn uploaded patch set 35 CR -1 V +1 06/05/24, 09:39

  20. core-ci voted on patch set 35 V +1 06/05/24, 09:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58733

  21. Christian Kuhn voted on patch set 35 CR +1 V +1 06/05/24, 09:50

  22. Stefan Bürk voted on patch set 35 CR +1 V +1 06/05/24, 09:51

  23. Garvin Hicking commented on patch set 35 CR +1 V +1 06/05/24, 09:52

    Thank you for this near-rewrite and taking me along the ride. Looks good from my POV!
  24. Christian Kuhn voted on patch set 35 CR +2 V +2 06/05/24, 10:07

  25. Christian Kuhn merged patch set 36 06/05/24, 10:07