82633[BUGFIX] Add constant evaluation in TypoScript functions
Commit message
Review
-
Code review CR +2 Done
- Christian Kuhn +2
- Garvin Hicking +1
- Stefan Bürk +1
-
Verified V +2 Done
- Christian Kuhn +2
- Garvin Hicking +1
- Stefan Bürk +1
- core-ci +1 CI
Issues 3
Files 11
- typo3/sysext/core/Classes/TypoScript/AST/Event/EvaluateModifierFunctionEvent.php+3 −2
- typo3/sysext/core/Classes/TypoScript/AST/AbstractAstBuilder.php+13 −9
- typo3/sysext/core/Classes/TypoScript/AST/AstBuilder.php+1 −1
- typo3/sysext/core/Classes/TypoScript/AST/CommentAwareAstBuilder.php+15 −3
- typo3/sysext/core/Classes/TypoScript/Tokenizer/Line/IdentifierFunctionLine.php+9 −8
- typo3/sysext/core/Classes/TypoScript/Tokenizer/LosslessTokenizer.php2+17 −19
- typo3/sysext/core/Classes/TypoScript/Tokenizer/LossyTokenizer.php9+10 −11
- typo3/sysext/core/Tests/Functional/Fixtures/Extensions/test_typoscript_ast_function_event/Classes/EventListener/TyposcriptTestFunction.php+1 −1
- typo3/sysext/core/Tests/Unit/TypoScript/AST/AstBuilderInterfaceTest.php2+157 −0
- typo3/sysext/core/Tests/Unit/TypoScript/Tokenizer/Line/IdentifierFunctionLineTest.php+4 −4
- typo3/sysext/core/Tests/Unit/TypoScript/Tokenizer/TokenizerInterfaceTest.php6+156 −20
Review log 6095
Show 4370 earlier messages
-
Garvin Hicking uploaded patch set 1 01/24/24, 12:45
-
Garvin Hicking marked it work in progress 01/24/24, 12:46
-
core-ci voted on patch set 1 V -1 01/24/24, 12:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52078
-
Garvin Hicking uploaded patch set 2
V -101/24/24, 15:25Implement CGL -
core-ci voted on patch set 2 V -1 01/24/24, 15:33 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52092
-
Garvin Hicking uploaded patch set 3
V -101/24/24, 16:06Address bad node getValue -
core-ci voted on patch set 3 V -1 01/24/24, 16:12 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52097
-
Garvin Hicking uploaded patch set 4
V -101/24/24, 20:46Adress test errors -
core-ci voted on patch set 4 V -1 01/24/24, 20:51 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52110
-
Garvin Hicking uploaded patch set 5
V -101/24/24, 20:54Adress duplicate test names -
core-ci voted on patch set 5 V +1 01/24/24, 21:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52111
-
Garvin Hicking uploaded patch set 6
V +101/25/24, 09:36Rework central parts add tests -
core-ci voted on patch set 6 V +1 01/25/24, 09:41 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52143
-
Garvin Hicking updated the commit message in patch set 7
V +101/25/24, 09:45Adjust commit message -
Garvin Hicking marked it ready for review 01/25/24, 09:45
-
core-ci voted on patch set 7 V +1 01/25/24, 09:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52144
-
Garvin Hicking uploaded patch set 8
V +101/26/24, 06:22Bind to proper ticket -
core-ci voted on patch set 8 V +1 01/26/24, 06:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/52211
-
Christian Kuhn commented on patch set 8 02/09/24, 11:21
LossyTokenizer.php:33please use TokenStreamInterface for type hints everywhere, that's what it is for. the single one place where you need to have that distinction to the comment-aware implemenation in the ast builder, we use the instanceof check and then we know.LossyTokenizer.php:590no we don't. addToList({$my,constant},42) will use the constant {$my,constant}. i don't see the point why you start looking at function value separators. the function tokenizer must not know about implementation details of single function value separators.LossyTokenizer.php:635please either keep the signature as is and always work on $this properties, OR change the tokenizer to always receive all arguments and never work on $this properties (this could be hard, there was a reason I went with the properties). But please to NOT implement a mixture of both.TokenizerInterfaceTest.php:3175this is broken. 17 is the correct value.TokenizerInterfaceTest.php:318317 -
Garvin Hicking uploaded patch set 9
V +102/12/24, 14:56 -
core-ci voted on patch set 9 V -1 02/12/24, 15:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53086
-
Garvin Hicking uploaded patch set 10
V -102/12/24, 15:21 -
core-ci voted on patch set 10 V -1 02/12/24, 15:26 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53088
-
Garvin Hicking uploaded patch set 11
V -102/12/24, 15:27 -
core-ci voted on patch set 11 V -1 02/12/24, 15:36 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53093
-
Garvin Hicking uploaded patch set 12
V -102/12/24, 15:42 -
core-ci voted on patch set 12 V +1 02/12/24, 15:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53095
-
Garvin Hicking commented on patch set 12 02/12/24, 16:16
LossyTokenizer.php:33ReplyAck. Totally went off-rails there, don't know why I didn't find this in first place.LossyTokenizer.php:590ReplyI see; I made it more complicated because I figured we should tokenize each function argument here too to access them one-by-one. I've now reworked it to just treat everything between "(" and ")" as a single function value now. (Note that something like `prependString(()` probably won't work (and doesn't now) with the simple parsing approach, if you wanted to use the closing parantheses character at any point within the function argument list. And we don't have string enclosure characters like " or ' in the TypoScript "specs" here to prevent it)LossyTokenizer.php:635ReplyThis is the main pain point here. I've now reworked it to take mandatory method arguments now that indicate the "working scope" of the method. My current approach starts another TokenStream here, so that it can use the "bait and switch" method of how parseValueForConstances replaces a stream. Since I cannot replace the WHOLE stream, but just the function argument stream part of it, I needed a method to only work on that subpart. The subpart can't be another instance of Lossy/lessTokenizer to allow operating on `$this` because that would break the whole idea of a single Tokenizer service. To prevent needing a whole rework of more "DTOs" for a stream and refactoring the whole method, I tried to go a way with "least breakage" here. This is, BTW, the reason why on the places below, the context shifts from column "17" to a relative "7" one. When parsing "addToList(XXX)", it starts at character position 10 to parse XXX. The new substream starts with a 0-index (because "currentLineString" will be set to the function argument list, starting at 0). I do think we might want to look into this together some more, maybe you have another approach on how to solve the "branching off" of the function parsing here, so that a distinct stream can come out of it which binds the constants to that stream in the AST later on. -
Christian Kuhn commented on patch set 12 02/13/24, 08:59
LossyTokenizer.php:590Replygood point. we may need to allow quoting `)` as `\)` when that char is used as function value. The 'identifier' parsing does a similar thing for `.` (most commonly used in flex form details). should of course be a separate patch.LossyTokenizer.php:635Replyyeah. i'm aware that handing over the subject data as argument destroys the character position of subsequent tokens. that's why i went with $this state in this method in the first place. i don't like it either. i guess we have to count the chars of the result or something to manipulate the char-counting for the tokens after the value stream. -
Garvin Hicking uploaded patch set 13
V +102/26/24, 08:52Adjust phpunit attributes resolve merge conflict -
core-ci voted on patch set 13 V -1 02/26/24, 08:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53587
-
Garvin Hicking uploaded patch set 14
V -103/05/24, 10:32 -
core-ci voted on patch set 14 V +1 03/05/24, 10:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53929
-
Simon Schaufelberger commented on patch set 14 03/05/24, 10:54
#102913 is not accessible. Why? -
Garvin Hicking commented on patch set 14 03/05/24, 18:53
Reply
Just to reduce mailspam. This is WIP, the ticket was just a stub. I did not want to flood the actual ticket # for this with WIP gerrit pushes - but now I see, the "related" line does trigger this. So this was a moot issue. I've unprivated the issue. ;) -
Simon Schaufelberger wrote 03/05/24, 19:10
Removed cc Simon Schaufelberger. -
Garvin Hicking uploaded patch set 15
V +103/06/24, 08:17 -
core-ci voted on patch set 15 V -1 03/06/24, 08:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/53998
-
Garvin Hicking uploaded patch set 16
V -103/06/24, 16:14Fix all the things -
Garvin Hicking commented on patch set 16 03/06/24, 16:16
LossyTokenizer.php:635ReplyI have added an offset to this: ``` $this->parseValueForConstants($functionBodyPart, $functionValueStream, $functionBodyStartPosition); ... $valueToken = new Token(TokenType::T_VALUE, $currentString, $this->currentLineNumber, $this->currentColumnInLine + $lastTokenEndPosition + $tokenOffsetPosition); ```TokenizerInterfaceTest.php:3175ReplyYes, reverted this line change and now uses offsets.TokenizerInterfaceTest.php:3183ReplyDone -
core-ci voted on patch set 16 V -1 03/06/24, 16:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54050
-
Garvin Hicking uploaded patch set 17
V -103/06/24, 16:39Fix CGL -
core-ci voted on patch set 17 V +1 03/06/24, 16:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54056
-
Marvin Buchmann voted on patch set 17 V +1 03/08/24, 09:24
-
Chris Müller uploaded patch set 18
V +105/02/24, 17:55 -
core-ci voted on patch set 18 V +1 05/02/24, 18:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57111
-
Chris Müller rebased onto patch set 19
V +105/28/24, 11:04 -
core-ci voted on patch set 19 V +1 05/28/24, 11:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58393
-
Christian Kuhn uploaded patch set 20
V +105/28/24, 11:32 -
Christian Kuhn commented on patch set 20 05/28/24, 11:33
cosmetics and un-removing 2 test files that should stay. -
Christian Kuhn commented on patch set 20 05/28/24, 11:36
TokenizerInterfaceTest.php:3573i think we should add a couple of tests to verify/show that functions with constants create proper token streams. -
core-ci voted on patch set 20 V +1 05/28/24, 11:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58394
-
Christian Kuhn uploaded patch set 21
V +105/29/24, 08:55 -
core-ci voted on patch set 21 V +1 05/29/24, 09:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58440
-
Christian Kuhn uploaded patch set 22
V +106/04/24, 08:21 -
Christian Kuhn commented on patch set 22 06/04/24, 08:22
TokenizerInterfaceTest.php:3573ReplyDone -
core-ci voted on patch set 22 V +1 06/04/24, 08:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58667
-
Christian Kuhn commented on patch set 22 06/04/24, 08:44
LosslessTokenizer.php:859i wonder if we should just always set the stream and except in the line getter if it's null. this could simplify the ast builders a bit. -
Christian Kuhn uploaded patch set 23
V +106/04/24, 09:54 -
core-ci voted on patch set 23 V -1 06/04/24, 10:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58674
-
Christian Kuhn uploaded patch set 24
V -106/04/24, 10:31 -
Christian Kuhn commented on patch set 23 06/04/24, 10:31
LosslessTokenizer.php:859ReplyDone -
core-ci voted on patch set 24 V +1 06/04/24, 10:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58676
-
Christian Kuhn uploaded patch set 25
V +106/04/24, 12:38 -
Christian Kuhn uploaded patch set 26 06/04/24, 12:41
-
core-ci voted on patch set 25 V +1 06/04/24, 12:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58679
-
core-ci voted on patch set 26 V +1 06/04/24, 12:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58680
-
Christian Kuhn uploaded patch set 27
V +106/04/24, 13:44 -
core-ci voted on patch set 27 V +1 06/04/24, 13:49 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58681
-
Christian Kuhn uploaded patch set 28
V +106/04/24, 16:25
-
Christian Kuhn commented on patch set 28 06/04/24, 16:26
rebased on top of https://review.typo3.org/c/Packages/TYPO3.CMS/+/84491 -
core-ci voted on patch set 28 V +1 06/04/24, 16:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58688
-
Christian Kuhn uploaded patch set 29
V +106/04/24, 18:31 -
Christian Kuhn commented on patch set 29 06/04/24, 18:31
-
core-ci voted on patch set 29 V +1 06/04/24, 18:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58699
-
Christian Kuhn commented on patch set 29 CR -1 06/04/24, 18:39
AstBuilderInterfaceTest.php:2815should assert ast, not its array representation. -
Christian Kuhn uploaded patch set 30
CR -1V +106/05/24, 07:20 -
Christian Kuhn commented on patch set 30 06/05/24, 07:21
AstBuilderInterfaceTest.php:2815ReplyDone -
core-ci voted on patch set 30 V +1 06/05/24, 07:24 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58722
-
Christian Kuhn uploaded patch set 31
V +106/05/24, 07:33 -
core-ci voted on patch set 31 V +1 06/05/24, 07:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58724
-
Christian Kuhn uploaded patch set 32
V +106/05/24, 07:41 -
Christian Kuhn uploaded patch set 33 06/05/24, 07:43
-
core-ci voted on patch set 32 V +1 06/05/24, 07:47 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58725
-
core-ci voted on patch set 33 V +1 06/05/24, 07:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58727
-
Christian Kuhn updated the commit message in patch set 34
V +106/05/24, 07:51 -
core-ci voted on patch set 34 V +1 06/05/24, 07:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58728
-
Christian Kuhn commented on patch set 34 CR -1 06/05/24, 09:21
looking at 2 minor details. -
Christian Kuhn uploaded patch set 35
CR -1V +106/05/24, 09:39 -
core-ci voted on patch set 35 V +1 06/05/24, 09:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/58733
-
Christian Kuhn voted on patch set 35 CR +1 V +1 06/05/24, 09:50
-
Stefan Bürk voted on patch set 35 CR +1 V +1 06/05/24, 09:51
-
Garvin Hicking commented on patch set 35 CR +1 V +1 06/05/24, 09:52
Thank you for this near-rewrite and taking me along the ride. Looks good from my POV! -
Christian Kuhn voted on patch set 35 CR +2 V +2 06/05/24, 10:07
-
Christian Kuhn merged patch set 36 06/05/24, 10:07