Review main Merged

83747[FEATURE] Add record-based transformation Data Processor

Benni Mack CR +2 V +2 Expert +1997 −27 updated 05/14/24, 14:40

Commit message

This patch consists of three aspects:

1. Record Object (internal)

A new value object for records, which contains only the fields relevant
for this specific Content Type. It decorates another RawRecord which
holds the original raw database record. The RecordInterface, which both
implement, allows for unified usage.

2. RecordFactory (internal)

The factory for the Record object. Takes the table name and a raw record
as input and builds the Record object from it. This class contains three
more public methods, which relate to TCA schema. These will be moved to
an appropriate place as soon as we have TCA schema at home.

3. RecordTransformationDataProcessor (feature)

The actual feature of this patch. The new DataProcessor allows to
utilize the Record object inside Fluid templates (only). It can be used
in various ways. Most prominent use cases are for chaining the
DatabaseQueryProcessor or transforming the current {data} variable of
FLUIDTEMPLATE views. The PAGEVIEW does have an enhanced PageInformation
object already.

Example DatabaseQueryProcessor:

  page.10.dataProcessing.10 = database-query
  page.10.dataProcessing.10.as = mainContent
  page.10.dataProcessing.10.table = tt_content
  page.10.dataProcessing.10.select.where = colPos=0
  page.10.dataProcessing.10.dataProcessing.10 = record-transformation

Example FLUIDTEMPLATE:

    tt_content.text {
      templateName = Text
      dataProcessing {
        10 = record-transformation
        10 {
          as = data
        }
      }
    }

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Nikita Hovratov +1
    • Andreas Kienast +1
  • Verified V +2 Done
    • Benni Mack +2
    • Nikita Hovratov +1
    • Andreas Kienast +1
    • core-ci +1 CI

Issues 1

Files 24

Show 4 more files

Review log 70105

Show 5180 earlier messages
  1. Benni Mack uploaded patch set 1 04/11/24, 13:10

  2. core-ci voted on patch set 1 V -1 04/11/24, 13:17 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55771

  3. Benni Mack uploaded patch set 2 V -1 04/11/24, 14:09

  4. core-ci voted on patch set 2 V -1 04/11/24, 14:16 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55775

  5. Benni Mack uploaded patch set 3 V -1 04/11/24, 14:51

  6. core-ci voted on patch set 3 V -1 04/11/24, 14:57 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55776

  7. Benni Mack uploaded patch set 4 V -1 04/11/24, 15:03

  8. core-ci voted on patch set 4 V -1 04/11/24, 15:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55777

  9. Benni Mack uploaded patch set 5 V -1 04/11/24, 16:51

  10. core-ci voted on patch set 5 V -1 04/11/24, 16:58 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55782

  11. Nikita Hovratov rebased onto patch set 6 V -1 04/11/24, 20:34

  12. core-ci voted on patch set 6 V -1 04/11/24, 20:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55790

  13. Stefan Bürk uploaded patch set 7 V -1 04/12/24, 11:34

  14. core-ci voted on patch set 7 V +1 04/12/24, 11:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/55860

  15. Stefan Bürk rebased onto patch set 8 V +1 04/26/24, 19:43

  16. core-ci voted on patch set 8 V +1 04/26/24, 19:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/56801

  17. Nikita Hovratov uploaded patch set 9 04/26/24, 19:57

    Copied Votes: * Verified+1 (copy condition: "**changekind:NO_CHANGE**")
  18. core-ci voted on patch set 9 V +1 04/26/24, 20:03 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/56803

  19. Benni Mack uploaded patch set 10 V +1 05/02/24, 20:07

  20. core-ci voted on patch set 10 V +1 05/02/24, 20:12 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57114

  21. Benni Mack uploaded patch set 11 V +1 05/02/24, 20:21

  22. core-ci voted on patch set 11 V +1 05/02/24, 20:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57115

  23. Oli Bartsch uploaded patch set 12 V +1 05/03/24, 11:19

  24. core-ci voted on patch set 12 V -1 05/03/24, 11:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57171

  25. Oli Bartsch uploaded patch set 13 V -1 05/03/24, 16:48

  26. core-ci voted on patch set 13 V +1 05/03/24, 16:53 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57175

  27. Benni Mack uploaded patch set 14 V +1 05/07/24, 20:23

  28. core-ci voted on patch set 14 V +1 05/07/24, 20:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57362

  29. Simon Schaufelberger commented on patch set 14 05/07/24, 23:07

    Feature-103783-RecordTransformationDataProcessor.rst:20

    space missing
  30. Benni Mack commented on patch set 15 V +1 05/08/24, 06:24

    Published edit on patch set 14.
  31. Benni Mack commented on patch set 15 05/08/24, 06:24

    Feature-103783-RecordTransformationDataProcessor.rst:20 Reply

    Done
  32. core-ci voted on patch set 15 V +1 05/08/24, 06:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57387

  33. Simon Schaufelberger wrote 05/08/24, 07:50

    Removed cc Simon Schaufelberger.
  34. Simon Praetorius commented on patch set 15 05/08/24, 09:38

    Great effort to introduce value objects here. 😊

    RecordFactory.php:41

    Should this maybe throw an exception if the table does not exist in TCA?

    RecordFactory.php:59

    Aren't these two variables the same?

    RecordFactory.php:64

    I guess this covers type definitions that refer to fields that are not properly defined as columns in TCA? If so, maybe add a comment to explain this.

    RecordFactory.php:66

    I don't get the first part of this condition: Should other arbitrary fields be allowed that are not defined in TCA? Doesn't seem right to me.

    RecordFactory.php:117

    Is it intended for the timestamps to be UTC? If not: (new \DateTimeImmutable())->setTimestamp(...)

    RecordFactory.php:179

    Is the last fallback correct here? I think it should be $tcaForTable['types'][0].

    RecordFactory.php:188

    This looks confusing without a comment explaining it. Maybe also extract it to a helper method that can be used below as well?

    RecordTransformationProcessor.php:71

    I'm not sure if this is helpful. There might be cases where data isn't an array containing the raw record.
  35. Georg Ringer commented on patch set 15 05/08/24, 15:21

    one small nitpicking: while the processor is called RecordTransformDataProcessor, the name is transform-records. singular vs plural and ordering. I would go with TransformRecordDataProcessor and transform-record
  36. Nikita Hovratov commented on patch set 15 05/08/24, 18:13

    RecordFactory.php:62

    This should be $typeFieldName !== null => the value could also be "0" but with an existing type field.
  37. Nikita Hovratov commented on patch set 15 05/08/24, 18:30

    RecordFactory.php:124

    isset($properties[$tcaConfig['ctrl']['descriptionColumn'])
  38. Nikita Hovratov commented on patch set 15 05/08/24, 18:50

    RecordFactory.php:50

    I don't fully understand the reasoning behind this special treatment of tt_content. It only makes it harder to use in scenarios where you need the actual database table. E.g. if you want to render nested Content Elements you need the table name and the raw record: <f:cObject.render typoscriptObjectPath="{record.fullType}" table="{record.mainType}" data="{record.rawRecord}"/> Is there any plan to introduce the naming / fallback layer for "content" => "tt_content" in other places, too?
  39. Nikita Hovratov commented on patch set 15 05/08/24, 19:05

    RecordFactory.php:105

    Import FQCN

    RecordInterface.php:46

    It's not possible right now to use this in Fluid Context, is it? Maybe we could add a getter for the array representation? Could be useful if ViewHelper expect an array (for type hinted methods).

    Feature-103783-RecordTransformationDataProcessor.rst:50

    I think we also need some examples on how to use this variable in Fluid. The problem is, the debug output is not helpful for integrators and they will fall on their nose trying to retrieve {record.properties.xyz} because it is displayed in the var dump. See example how I did this for Content Blocks here: https://docs.typo3.org/p/contentblocks/content-blocks/main/en-us/Templating/Index.html#accessing-variables
  40. Nikita Hovratov uploaded patch set 16 V +1 05/09/24, 14:01

  41. Nikita Hovratov commented on patch set 16 05/09/24, 14:02

    RecordFactory.php:62 Reply

    Done

    RecordFactory.php:105 Reply

    Done

    RecordFactory.php:124 Reply

    Done
  42. core-ci voted on patch set 16 V +1 05/09/24, 14:05 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57445

  43. Nikita Hovratov uploaded patch set 17 V +1 05/09/24, 14:58

  44. Nikita Hovratov commented on patch set 17 05/09/24, 14:59

    RecordFactory.php:179 Reply

    Done
  45. core-ci voted on patch set 17 V +1 05/09/24, 15:05 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57446

  46. Nikita Hovratov uploaded patch set 18 V +1 05/09/24, 15:31

  47. Nikita Hovratov commented on patch set 17 05/09/24, 15:31

    RecordFactory.php:41 Reply

    Done

    RecordFactory.php:59 Reply

    Done

    RecordFactory.php:64 Reply

    Done

    RecordFactory.php:66 Reply

    Done

    RecordFactory.php:188 Reply

    Done
  48. core-ci voted on patch set 18 V +1 05/09/24, 15:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57447

  49. Nikita Hovratov uploaded patch set 19 V +1 05/09/24, 16:07

  50. core-ci voted on patch set 19 V -1 05/09/24, 16:14 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57448

  51. Nikita Hovratov uploaded patch set 20 V -1 05/09/24, 16:17

  52. core-ci voted on patch set 20 V -1 05/09/24, 16:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57449

  53. Nikita Hovratov uploaded patch set 21 V -1 05/09/24, 17:26

  54. core-ci voted on patch set 21 V +1 05/09/24, 17:31 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57450

  55. Nikita Hovratov commented on patch set 21 05/09/24, 17:34

    Record.php:37

    There is no way to access special properties like `isDisabled` or `publishAt` in Fluid yet. Should we define getter for them, too?

    Record.php:119

    This is either the original uid, when in workspace context or the overlaid uid when on a translation. The naming is hard here, because it is more usage related (uid used to later resolve relations, I guess). Maybe we should split this in `getOriginalUid` and `getLocalizedUid` for usages in Fluid?
  56. Nikita Hovratov commented on patch set 21 05/09/24, 17:35

    Reply

    Yes, this should be streamlined.
  57. Nikita Hovratov commented on patch set 21 05/09/24, 17:35

    Feature-103783-RecordTransformationDataProcessor.rst:50 Reply

    Done
  58. Nikita Hovratov commented on patch set 21 05/09/24, 17:45

    Record.php:77

    One problem that might pop up is, that "type" is a really common name. In most cases "type" is also used as the TCA `type` field, so this would result in the same output. However, for any other usage this would make it impossible to retrieve the property. In Content Blocks I named this `typeName`. Do you think we could stick to this name? This would also ease the transition to Content Blocks later.
  59. Nikita Hovratov uploaded patch set 22 V +1 05/09/24, 18:04

  60. core-ci voted on patch set 22 V +1 05/09/24, 18:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57451

  61. Nikita Hovratov uploaded patch set 23 V +1 05/09/24, 18:29

  62. core-ci voted on patch set 23 V +1 05/09/24, 18:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57452

  63. Nikita Hovratov commented on patch set 23 05/09/24, 18:36

    RecordTransformationProcessor.php:71 Reply

    Yeah, this will cause a bug when there is a field called `data` in the record. I changed things up a bit, so that the input variable name is now `data`. This works for FLUIDTEMPLATE and DataBaseQueryProcessor without causing side-effects. @benni@typo3.org I'm not sure if we actually should handle the previous default case "If variableName left empty, use all properties as input". I have no knowledge of such a case.
  64. Nikita Hovratov uploaded patch set 24 V +1 05/09/24, 18:36

  65. core-ci voted on patch set 24 V +1 05/09/24, 18:41 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57453

  66. Nikita Hovratov uploaded patch set 25 V +1 05/10/24, 13:16

  67. Nikita Hovratov commented on patch set 25 05/10/24, 13:18

    Record.php:77 Reply

    Done

    RecordFactory.php:50 Reply

    Done
  68. core-ci voted on patch set 25 V +1 05/10/24, 13:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57479

  69. Nikita Hovratov uploaded patch set 26 V +1 05/10/24, 13:31

  70. Nikita Hovratov commented on patch set 25 05/10/24, 13:33

    Reply

    I chose to change the identifier to record-transformation and keep the class name the same. This seems more in line with current naming without "verb-style" names. E.g. database-query vs. query-database.
  71. core-ci voted on patch set 26 V +1 05/10/24, 13:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57481

  72. Nikita Hovratov uploaded patch set 27 V +1 05/10/24, 13:48

  73. Nikita Hovratov commented on patch set 27 05/10/24, 13:49

    RecordFactory.php:117 Reply

    Done
  74. core-ci voted on patch set 27 V +1 05/10/24, 13:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57484

  75. Nikita Hovratov uploaded patch set 28 V +1 05/10/24, 15:58

  76. Nikita Hovratov commented on patch set 28 05/10/24, 15:59

    Record.php:37 Reply

    Done
  77. core-ci voted on patch set 28 V +1 05/10/24, 16:04 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57486

  78. Nikita Hovratov uploaded patch set 29 V +1 05/10/24, 16:49

  79. Nikita Hovratov commented on patch set 29 05/10/24, 16:50

    Record.php:119 Reply

    Done
  80. core-ci voted on patch set 29 V -1 05/10/24, 16:54 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57487

  1. Nikita Hovratov uploaded patch set 30 V -1 05/10/24, 17:10

  2. core-ci voted on patch set 30 V +1 05/10/24, 17:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57488

  3. Andreas Kienast commented on patch set 30 CR -1 05/13/24, 06:42

    RecordFactory.php:192

    The load of `unset()` calls clutters the code very much. IMO the same could be achieved by `$properties = array_diff_key($properties, $systemProperties);`. Unfortunately, removing the `unset()` calls apparently doesn't let RecordFactory related tests fail.
  4. Andreas Kienast commented on patch set 30 05/13/24, 07:13

    RecordFactory.php:230

    When creating a new record, `tstamp` matches `crdate`. Perhaps it makes sense to compare these two values and write `lastUpdatedAt` only if there is a difference.

    Feature-103783-RecordTransformationDataProcessor.rst:87

    Perhaps this is something `TYPO3\CMS\Extbase\Utility\DebuggerUtility` could deal with in the future(tm).

    RecordTransformationProcessor.php:56

    ): array
  5. Benni Mack uploaded patch set 31 CR -1 V +1 05/13/24, 19:50

  6. Benni Mack uploaded patch set 32 05/13/24, 19:52

  7. Benni Mack commented on patch set 32 05/13/24, 19:55

    RecordFactory.php:192 Reply

    that does not work unfortunately. The systemProperties also contains the "language" key (whereas the properties have l18n_parent or l10n_parent, depending on the configuration).

    RecordFactory.php:230 Reply

    currently this only reflects the current DB state, we could add a getter later-on IMHO

    RecordInterface.php:46 Reply

    let's handle this in the next patch of this line

    Feature-103783-RecordTransformationDataProcessor.rst:87 Reply

    yes. good follow up to spend some love to the DebuggerUtility

    RecordTransformationProcessor.php:56 Reply

    Done
  8. core-ci voted on patch set 31 V +1 05/13/24, 19:56 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57643

  9. core-ci voted on patch set 32 V +1 05/13/24, 19:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57644

  10. Nikita Hovratov voted on patch set 32 CR +1 V +1 05/13/24, 20:30

  11. Andreas Kienast commented on patch set 32 CR -1 05/14/24, 14:04

    Record.php:66

    Should this be `($this->systemProperties?->toArray() ?? [])` since `$this->systemProperties` is nullable?
  12. Nikita Hovratov uploaded patch set 33 CR +1 V +1 05/14/24, 14:09

  13. Nikita Hovratov commented on patch set 33 05/14/24, 14:09

    Record.php:66 Reply

    Done
  14. core-ci voted on patch set 33 V -1 05/14/24, 14:14 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57719

  15. Andreas Kienast commented on patch set 33 CR -1 05/14/24, 14:16

    Record.php:66 Reply

    You missed the parentheses. This will now trigger an exception: `Unsupported operand types: array + null`.
  16. Andreas Kienast uploaded patch set 34 05/14/24, 14:17

    Copied Votes: * Code-Review-1 (copy condition: "**changekind:NO_CHANGE** OR is:MIN") * Verified-1 (copy condition: "**changekind:NO_CHANGE**")
  17. Andreas Kienast uploaded patch set 35 CR -1 V -1 05/14/24, 14:18

  18. Andreas Kienast commented on patch set 35 05/14/24, 14:18

    Record.php:66 Reply

    Done
  19. Nikita Hovratov uploaded patch set 36 05/14/24, 14:18

  20. core-ci voted on patch set 35 V +1 05/14/24, 14:24 Copied votes on follow-up patch sets have been updated: * Verified+1 has been copied to patch set 36 (copy condition: "changekind:NO_CHANGE"). Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57721

  21. core-ci voted on patch set 36 V +1 05/14/24, 14:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/57722

  22. Andreas Kienast voted on patch set 36 CR +1 V +1 05/14/24, 14:35

  23. Nikita Hovratov voted on patch set 36 CR +1 V +1 05/14/24, 14:39

  24. Benni Mack commented on patch set 36 CR +2 V +2 05/14/24, 14:40

    Thanks!
  25. Benni Mack merged patch set 37 05/14/24, 14:40