Review main Abandoned

50086[FEATURE] Unify chaining and nesting of dataProcessors

Jigal van Hemert CR 0 V 0 Hard +941 −34 updated 11/09/21, 21:27

Commit message

Some dataProcesors support further processing for each object/record they
produce (nesting), others support using data from outside the 'data'
array (chaining).

For "nesting" it should only be documented that each record/object that
is generated is further processed by applying the sub-dataProcessing
setting.
The CommaSeparatedValueProcessor now supports "nesting".

For "chaining" a new property of each setting that is meant to configure
the input data is introduced:
source : can have the values
* "data": read property with the given name from the data array
* "field": read property with the given name from outside the data array
* "value": literal value, no property name

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 9

Review log 2530

Show 45 earlier messages
  1. Jigal van Hemert uploaded patch set 1 10/03/16, 20:53

  2. TYPO3com voted on patch set 1 V -1 10/03/16, 20:57 Completed build in 03m 17s on Mon, 3 Oct, 10:57 PM Test Summary: 30 of 18603 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3319

  3. Sascha Egerer commented on patch set 1 CR -1 10/04/16, 05:17

    I'm open for comments for my suggestion and will remove my -1 if you have good arguments ;-)

    CommaSeparatedValueProcessor.php:119

    IMHO We need a more generic way to do chaining. Implementing this in every DataProcessor is not an option for me. This can be done for exmple with a "seperate" forEachDataProcessor that iterates over a list and processes each item. This would make the TS configuration a little bit more complex as to need another level of configuration but this would be very flexible. Another option could be that we implement a new "DataSetProcessors" key in typoscript on the same level as "DataProcessors". This could be the same impementation as the ForEachProcessor but as a core component.
  4. Jigal van Hemert commented on patch set 1 CR -2 10/04/16, 05:43

    CSV processor is not backwards compatible now because of the nesting support. Need to figure out a way to have both.
  5. Jigal van Hemert commented on patch set 1 10/04/16, 14:37

    CommaSeparatedValueProcessor.php:119 Reply

    During T3DD16 we discussed this and we decided against creating a foreach processor. It would only complicate the configuration and not add value. The few lines of nesting "logic" would be moved to this foreach processor which is too much overhead.
  1. Jigal van Hemert uploaded patch set 2 10/05/16, 05:54

  2. TYPO3com voted on patch set 2 V -1 10/05/16, 05:58 Completed build in 03m 35s on Wed, 5 Oct, 07:58 AM Test Summary: 18603 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3324

  3. Jigal van Hemert uploaded patch set 3 10/05/16, 12:13

  4. TYPO3com voted on patch set 3 V +1 10/05/16, 12:17 Completed build in 03m 18s on Wed, 5 Oct, 02:17 PM Test Summary: 18603 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-3325

  5. Jigal van Hemert voted on patch set 3 CR 0 10/23/16, 18:38

  6. Anja Leichsenring commented on patch set 3 10/24/16, 13:00

    Looks good to me. Can you please add a Feature rst. Thanks.
  7. Daniel Alder commented on patch set 3 12/01/16, 16:32

    code looks good, but didn't tested functionality in files, gallery, split & databasequery yet. will do that, after some clarification from you ;)

    CommaSeparatedValueProcessor.php

    If I nest a 'CommaSeparatedValueProcessor' with a 'CommaSeparatedValueProcessor' I'am forced to set a 'fieldname' in the sub-processor. Since I am getting an indexed array I can only get specific entries, which makes no sense to me. so Sascha Egerer's idea of a 'forEach' processor would make sense... or could you provide an example, how I proceed in the sub processor? Beside: Processing the index '0' doesn't work, since it will be checked 'empty($fieldName)' Beside 2: I know, this is not cause of your changes, but using an empty 'fieldEnclosure' doesn't work
  8. Susi Moog commented on patch set 3 10/27/17, 10:41

    ping... @Jigal can you answer the questions?
  9. Jigal van Hemert commented on patch set 3 10/27/17, 21:45

    > Looks good to me. Can you please add a Feature rst. Thanks. Will add the rst!
  10. Jigal van Hemert commented on patch set 3 10/28/17, 09:27

    CommaSeparatedValueProcessor.php Reply

    Some people seem to have CSV data inside CSV fields. I haven't come across these yet, but the output of the CSV data with numeric keys in the array is not suitable yet for use in any other data processor. I think the solution would be to use a string prefix to the keys for the data that goes into the 'data' field (used for nesting). I'm not against a forEach processor, but it would add an extra layer in the configuration and it would not help with chaining data processors. Beside: I think the string prefix for nesting will help here Beside 2: I'll have a look, but it'll go into a separate patch as it's a bugfix
  11. Jigal van Hemert uploaded patch set 4 10/28/17, 19:47

  12. TYPO3com voted on patch set 4 V -1 10/28/17, 19:56 Completed build in 08m 57s on Sat, 28 Oct, 09:56 PM Test Summary: 138 of 57397 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-15898

  13. Jigal van Hemert uploaded patch set 5 10/28/17, 21:30

  14. TYPO3com voted on patch set 5 V +1 10/28/17, 21:39 Completed build in 09m 12s on Sat, 28 Oct, 11:39 PM Test Summary: 57038 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-15899

  15. Susi Moog commented on patch set 5 CR -2 11/29/17, 15:14

    Please talk to Benji about the concept for this as he has some valid concerns regarding the usability of this feature.
  16. Jigal van Hemert commented on patch set 5 11/29/17, 15:37

    > Please talk to Benji about the concept for this as he has some > valid concerns regarding the usability of this feature. I have some serious concerns regarding the usefulness of the current state of the dataProcessors. In order to create a gallery content element I had to use my own versions of several dataProcessors because they couldn't use data from a previous one. Please tell Benji that he is very welcome to explain a better concept to make dataProcessors more versatile in their use.
  17. Tymoteusz Motylewski commented on patch set 5 02/07/18, 16:07

    Hi Jigal, Can you explain the issue better? I feel that there is a real issue in TYPO3 in terms of flexibility. However I'm not seeing it from the forge issue description nor from patch. Can you maybe show the case which is hard to solve with current core, ideally with current code you use to solve it (so we can see the pain)?
  18. Jigal van Hemert commented on patch set 5 02/11/18, 15:29

    > Hi Jigal, > Can you explain the issue better? I feel that there is a real issue > in TYPO3 in terms of flexibility. However I'm not seeing it from > the forge issue description nor from patch. Hi Tymoteusz, > > Can you maybe show the case which is hard to solve with current > core, ideally with current code you use to solve it (so we can see > the pain)? The current data processors from the core have some limited support for nesting and chaining. Nesting: each object/record that is output by the data processor can be fed to another data processor. This happens for example with the SQL data processor where each record from the query can be processed further. For this feature the next data processor is configured as a sub-property of the main data processor. Chaining: all the data which is output by the data processor can be processed by a next data processor. All the data processors are configured in a list of data processor (on the same level in TS). A data processor fetches data from the data array and assigns it to a variable/field outside the data array after processing. The pain is that the current data processors support nesting or chaining for specific situations. For example, the GalleryProcessor has a filesProcessedDataKey property that defines which property outside the data array (the processed data) defines which files to use. It can't use a property from the data array or even a literal value. I had to make an image gallery from a field that was stored in a flexform. A flexform data processor was easily made, but the file data processor could only process data from the data array. So, I needed to make my own modified file data processor. Someone else had to process CSV data, but in this case some fields in the CSV data contained CSV data themselves (not very common, but I found several cases "in the wild"). Unfortunately it was not possible to process each field with another CSV data processor. A solution would be to allow each property that configures the input data to also specify where the data should come from: a field in the data array, a field outside the data array ("processed data") or even a literal value. This is why the sub-property "source" is introduced.
  19. Anja Leichsenring wrote 08/05/18, 11:57

    Removed reviewer Anja Leichsenring.
  20. Georg Ringer commented on patch set 5 11/08/18, 21:34

    @Jigal van Hemert: how to proceed?
  21. Susi Moog commented on patch set 5 CR 0 03/01/19, 18:13

    I'm removing my -2 here, I'd ask @BenjaminKott to add his concerns directly.
  22. Susi Moog wrote 03/07/19, 17:39

    Removed reviewer Susanne Moog.
  23. Benni Mack commented on patch set 5 04/06/20, 09:39

    @All: What's the status here?
  24. Christian Kuhn commented on patch set 5 11/09/21, 21:27

    Hey. I'm abandoning this change for the sake of a cleaner review system for now: Nothing happened here for quite a while, the change can be resurrected if work continues.
  25. Christian Kuhn abandoned the change 11/09/21, 21:27