Review main Merged Breaking

74987[!!!][FEATURE] New TypoScript parser

Christian Kuhn CR +2 V +2 Impossible +19074 −1 updated 07/04/22, 19:13

Commit message

This adds a rewritten TypoScript syntax parser to ext:core.

The heart of the existing parser dates back to Kasper's
initial commit in 2003 and most parts have never been
touched structurally. Even though various parts of
TypoScript became less important in the Frontend rendering
chain over time, it still plays a central point. Looking at
the given parser approach, it became clear the project can
benefit from a revamped parser based on modern PHP code.

Goals:
* A well structured, flexible and tested codebase.
* Fix long standing syntax shenanigans and limits of
  the current parser approach and make the syntax
  interpretation more robust and resilient.
* Have a better cache layer that kicks in more often
  to ultimately gain speed building the FE TypoScript.
* Allow improving the Backend "Template" module to
  show and analyze more TypoScript details.
* Have an object based structure for the final TypoScript
  tree, and allow an array export of it to keep backwards
  compatibility.

The new implementation aims to fully substitute the old
parser. This first patch however only brings the "library"
part of needed code changes: There is no usage of the new
parser whatsoever with this patch, except of course unit
and functional tests.

To understand the structure of the new parser, it might be
helpful to have a rough understanding of the old parser. We'll
look at the frontend parsing steps only, Backend usages in
Template module are similar, though.

When accessing a frontend page, TYPO3 at some points finds the
requested page uid. It then builds the "rootline" of page
records from the top page root node down to the requested
page. This creates a list of page database records, mountpoints
and similar are taken care of at this point, too.

This rootline is fed to the old TypoScript parser. The parser now
finds all relevant sys_template records attached to the rootline
pages. These are the "entry" TypoScript snippets to parse. It
then resolves all "includes": Snippets from sys_template
include_static_file db records, snippets from various globals,
snippets from sys_template basedOn, and so on. It also resolves
@import and <INCLUDE_TYPOSCRIPT: syntax and substitutes them
with the included content. All that is gathered in one huge
string. The main parsing process then goes through this string
and creates the TypoScript array, while taking care of conditions
and constants substitution at the same time.

This approach has various drawbacks: First, gathering everything
into one main string forces to parse the entire string a-new for
each and every page: A different page could have different
includes or sys_template records attached. Secondly, parsing
conditions and constants while creating the final array from the
source string, ties "runtime" information (especially conditions)
hard into the main parsing process, rendering an effective cache
layer impossible. Third, the main strategy of gathering everything
in one string leads to various funny details, for instance when
opening a condition in one file and not closing it with [end] or
[global], this state leaks to the next file. Bracket handling "}"
has similar issues.

These structural issues can't be changed by refactoring the
given codebase, the only option is to rewrite the entire thing.

IncludeTree:
The entry point to the new parser is IncludeTree/TreeBuilder:
This one again receives the pages "rootline" from Frontend. But
instead of creating one huge string, it creates an object tree of
includes. Each attached sys_template record is a child of
"RootInclude" in the tree, and each sys_template include node
can have children for further includes (like include_static_file).
Single source snippets of each include are tokenized (see below)
and analyzed for @import and <INCLUDE_TYPOSCRIPT: and conditions:
If they exist, an include is marked as "split". Each part is then
represented by child includes. We thus receive a tree of objects
where imports are already resolved, and conditions are child nodes
in the tree.

The main advantage is this tree does not carry runtime information.
A single snippet *always* leads to the same tree, no matter if a
condition matched or not. This way, the tree can be cached and the
full tree, or parts of it can be re-used when requesting a
different page. To do that efficiently, a new cache "typoscript"
is established to store a serialized representation of the tree
as php file, which can be unserialized relatively quickly.

Tokenizer:
The tokenizer takes a single source snipped and creates a stream
of Line objects from it, with tokens representing the source line.
For instance, a TypoScript snipped like "foo = fooValue" creates
a IdentifierAssignmentLine object, with a T_IDENTIFIER token for
"foo" and a T_VALUE token for "fooValue". There are Line classes
for all the different Lines (assignment, copy, condition, ...) and
Tokens for the various details (T_VALUE, T_CONSTANT, ...). Tokens
are sometimes encapsulated in TokenStreams, for example multiple
identifiers "foo.bar = barValue" create a stream of the two
T_IDENTIFIER tokens "foo" and "bar". This depends on the line type.

We also have two different tokenizers: The Frontend uses the
LossyTokenizer which creates a stream only of "relevant" tokens:
Empty lines, comments and lines with invalid syntax are ignored.
The LosslessTokenizer however creates a 1:1 representation of the
source snipped, including token positions (line and start column).
This allows detailed analysis in the Backend Template module.

AstBuilder:
This is the third part of the structure: A representation of
parsed TypoScript as object tree. This is similar to the array
we're dealing with now, but it can carry additional functionality.
For instance, the well-known TypoScript array can be extracted
from it.
Using the AstBuilder works like this: Create the IncludeTree
first (from cache). Next, traverse the IncludeTree with a visitor
that looks at condition includes and sets an information
if they match. Then, feed all includes that matched to the
AstBuilder to create the object tree. The AstBuilder receives
a LineStream of each include and extends the object tree depending
on the line type.

This means: IncludeTree building and source tokenizing can be
cached in Frontend, applying constants and condition verdicts
together with building the finally TypoScript object tree is a
runtime step on each page. Separating tokenizing and AST building
with a cache layer in between is roughly twice as fast in
Frontend compared to the previous solution.

The patch establishes these three main structures and comes with
an extensive set of unit and functional tests. It is an extract
of a bigger WIP patch that already contains usages of the new
structure in Frontend and some parts of the Backend. The patch
also adds a ReST file to explain some subtle TypoScript syntax
changes. These will kick in as soon as we start using the
structure with upcoming changes.

The new parser should be rather robust already. Further changes
that start using the new parser will probably only change minor
things. For now, the entire structure is marked @internal,
though: API is still the array representation, only. This will
change later when we actively start using the TypoScript object
tree in the core.

Review

  • Code review CR +2 Done
    • Christian Kuhn +2
    • Stefan Bürk +1
    • Benni Mack +1
  • Verified V +2 Done
    • Christian Kuhn +2
    • Stefan Bürk +1
    • Benni Mack +1
    • core-ci +1 CI

Issues 5

Files 165

Show 145 more files

Review log 2333

Show 48 earlier messages
  1. Christian Kuhn uploaded patch set 1 06/27/22, 15:22

  2. core-ci voted on patch set 1 V +1 06/27/22, 15:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28439

  3. Christian Kuhn uploaded patch set 2 06/27/22, 17:10

  4. Christian Kuhn updated the commit message in patch set 3 06/27/22, 17:17

  5. core-ci voted on patch set 2 V +1 06/27/22, 17:17 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28441

  6. core-ci voted on patch set 3 V +1 06/27/22, 17:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28443

  7. Christian Kuhn uploaded patch set 4 06/27/22, 18:27

  8. core-ci voted on patch set 4 V +1 06/27/22, 18:34 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28444

  1. Christian Kuhn uploaded patch set 5 06/28/22, 13:36

  2. core-ci voted on patch set 5 V +1 06/28/22, 13:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28458

  3. Christian Kuhn uploaded patch set 6 06/28/22, 15:37

  4. core-ci voted on patch set 6 V +1 06/28/22, 15:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28460

  5. Christian Kuhn uploaded patch set 7 06/28/22, 16:37

  6. core-ci voted on patch set 7 V +1 06/28/22, 16:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28463

  7. Simon Schaufelberger updated the commit message in patch set 8 06/28/22, 18:28

  8. core-ci voted on patch set 8 V +1 06/28/22, 18:36 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28464

  9. Stefan Bürk commented on patch set 8 06/28/22, 20:12

    CurrentObjectPathStack.php:38

    Nitpick: superflous space ? Normally inc/decc are directly "attached" to the variable .. curious, not cgl rule for this.

    CurrentObjectPathStack.php:50

    Nitpick: superflous space ? Normally inc/decc are directly "attached" to the variable .. curious, not cgl rule for this.

    AbstractChildNode.php:58

    Are there not tests for it ? e.g. https://review.typo3.org/c/Packages/TYPO3.CMS/+/74987/7/typo3/sysext/core/Tests/Unit/TypoScript/AST/AstBuilderTest.php#1362 or are meant the final "replacement" within ContentObjectRenderer cObjGetSingle() ?

    AbstractInclude.php:45

    Should we make this method "final" ? Extending classes can handle differences in calles `serialize()`

    RootInclude.php:33

    NoOp - what speak against throwing a exception instead if "someone" tries it (later when we directly use object structure). Or do I oversee something ?

    RootInclude.php:38

    See above for setIdentifiert()

    basedOnAfterIncludeStatic.csv:6

    2x field "sorting" for sys_template ? Would bet this would fail on postgress, if test would test for a sorting on the same pid. Basicly wrong, even if not make a different for current case, because both templates are on different pids and thus sorting to "relevant".

    basedOnBeforeIncludeStatic.csv:6

    2x field "sorting" for sys_template ? Would bet this would fail on postgress, if test would test for a sorting on the same pid. Basicly wrong, even if not make a different for current case, because both templates are on different pids and thus sorting to "relevant".

    basedOnSimple.csv:6

    2x field "sorting" for sys_template ? Would bet this would fail on postgress, if test would test for a sorting on the same pid. Basicly wrong, even if not make a different for current case, because both templates are on different pids and thus sorting to "relevant".

    singleRootTemplate.csv:5

    2x field "sorting" for sys_template ? Would bet this would fail on postgress, if test would test for a sorting on the same pid. Basicly wrong, even if not make a different for current case, because only one template.

    twoPagesTwoTemplates.csv:6

    2x field "sorting" for sys_template ? Would bet this would fail on postgress, if test would test for a sorting on the same pid. Basicly wrong, even if not make a different for current case, because both templates are on different pids and thus sorting to "relevant".

    twoPagesTwoTemplatesBothClear.csv:6

    2x field "sorting" for sys_template ? Would bet this would fail on postgress, if test would test for a sorting on the same pid. Basicly wrong, even if not make a different for current case, because both templates are on different pids and thus sorting to "relevant".
  10. Markus Klein commented on patch set 8 06/29/22, 06:53

    Feature-97816-TypoScriptSyntaxImprovements.rst:68

    While we are at it. This line and line 52 above are very confusing for a newcomer. Why does the one qualify as comment there but not here. I would strongly suggestion to deprecate in-line comments completely!
  11. Stefan Bürk commented on patch set 8 06/29/22, 07:01

    Feature-97816-TypoScriptSyntaxImprovements.rst:68 Reply

    I would guess, because for "assignements" there must be e.g the ability for the field fallback handling: ``` page.10.field = nav_title // title ``` thus the reasin it is not a comment for assignment operator. That is a old double usage making line comment for assignment inpossible, but in generall possible for all other cases.
  12. Markus Klein commented on patch set 8 06/29/22, 08:59

    Feature-97816-TypoScriptSyntaxImprovements.rst:68 Reply

    I know that, no worries. My comment is about *new TYPO3 users*, not knowing such nasty details. I would take the chance now and make the hard cut and even disallow comments in the same line completely. There is no point about doing this, just put it in the line above. It makes everything easier to understand.
  13. Simon Schaufelberger commented on patch set 8 06/29/22, 10:50

    Feature-97816-TypoScriptSyntaxImprovements.rst:68 Reply

    > It makes everything easier to understand while typoscript is line based, this makes total sense in contrast to php which you could put more or less all in one line and it still works. I also give +1 for dropping inline comments.
  14. Christian Kuhn commented on patch set 8 07/01/22, 09:50

    CurrentObjectPathStack.php:38 Reply

    afaik no hard rule for that and i probably even mix it within the patch. i must say that i don't care too much for this detail. if we feel lucky, we could sort out details later and then have a global core patch to fix it to whatever we agreed on? -> i'll not change here now.

    CurrentObjectPathStack.php:50 Reply

    Ack

    AbstractChildNode.php:58 Reply

    Done

    AbstractInclude.php:45 Reply

    good idea.

    RootInclude.php:33 Reply

    yep. done.

    RootInclude.php:38 Reply

    yep. done.

    AbstractTokenStream.php:51 Reply

    Done

    Feature-97816-TypoScriptSyntaxImprovements.rst:68 Reply

    I tend to give integrators as much comment freedom as possible. Goal of my implementation was to especially fix the nasty '*/ has to be on a single line'. It's basically like that now: place comments anywhere (and it *is* helpful to be able to place a comment at the end of a condition line or similar), but be aware that assignments terminate at end of line: comment and other special chars are never interpreted, but have the advantage those need no quoting. This looks logical to me and is rather easy to explain. We could discuss a deprecation of the comment logic after main merge again, but it feels to me as if we should rather go with more freedom as proposed instead of more restrictions.

    basedOnAfterIncludeStatic.csv:6 Reply

    Done

    basedOnBeforeIncludeStatic.csv:6 Reply

    Done

    basedOnSimple.csv:6 Reply

    Done

    basedOnSimple.csv:6 Reply

    Done

    singleRootTemplate.csv:5 Reply

    Done

    twoPagesTwoTemplates.csv:6 Reply

    Done

    twoPagesTwoTemplatesBothClear.csv:6 Reply

    thank you! hope i got them all, now.
  15. Christian Kuhn uploaded patch set 9 07/01/22, 09:51

  16. core-ci voted on patch set 9 V -1 07/01/22, 10:00 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28561

  17. Christian Kuhn uploaded patch set 10 07/01/22, 11:02

  18. core-ci voted on patch set 10 V +1 07/01/22, 11:10 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/28565

  19. Simon Schaufelberger wrote 07/01/22, 12:32

    Removed cc Simon Schaufelberger.
  20. Benni Mack commented on patch set 10 V +1 CR +1 07/04/22, 07:19

    Feature-97816-TypoScriptSyntaxImprovements.rst:68 Reply

    Yes. I am in favor of allowing the "// my comment" only on separate lines and deprecating the existing behaviour (in a separate patch)
  21. Stefan Bürk voted on patch set 10 V +1 CR +1 07/04/22, 07:33

  22. Christian Kuhn voted on patch set 10 V +2 CR +2 07/04/22, 11:22

  23. Christian Kuhn merged patch set 11 07/04/22, 11:22

  24. Benni Mack commented on patch set 11 07/04/22, 11:31

    Thanks for the hard work on this!
  25. Stefan Bürk commented on patch set 11 07/04/22, 19:13

    Reply

    Same from me. Thanks for the hard work on this, covered with tests. Nice developed.