Review main Merged Breaking

90405[!!!][TASK] Use Record API in List Module

Benni Mack CR +2 V +2 Hard +484 −248 updated 09/02/25, 13:50

Commit message

List module is a perfect candidate to use Record API internally
to avoid access to arrays all the time.

Various public method are now strict-typed and use Record API,
which is why this patch is marked as breaking.

This change enables further patches to
- extract getQueryBuilder() in its separate class
- modernize the events and code base to deal with objects instead of
  strings and arrays
- reduce usages of BackendUtility for rendering values of records in the Backend

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Oli Bartsch +1
    • Stefan Bürk +1
  • Verified V +2 Done
    • Benni Mack +2
    • Oli Bartsch +1
    • Stefan Bürk +1
    • core-ci +1 CI

Issues 1

Files 10

Review log 3448

Show 1523 earlier messages
  1. Benni Mack uploaded patch set 1 08/15/25, 09:10

  2. core-ci voted on patch set 1 V -1 08/15/25, 09:17 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/83443

  3. Benni Mack uploaded patch set 2 V -1 08/15/25, 12:26

  4. core-ci voted on patch set 2 V -1 08/15/25, 12:40 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/83454

  5. Benni Mack uploaded patch set 3 V -1 08/15/25, 12:49

  6. core-ci voted on patch set 3 V +1 08/15/25, 12:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/83456

  7. Benni Mack updated the commit message in patch set 4 V +1 08/17/25, 20:31

  8. core-ci voted on patch set 4 V +1 08/17/25, 20:38 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/83538

  9. Garvin Hicking commented on patch set 4 08/17/25, 21:07

    DatabaseRecordList.php:654

    Wouldn't this cause a lot of DB>PHP traffic and I/O? If recordlists e.g. contain larger JSON data (eg maps, geo shapes) or binary data...? Using eager loading on a single record probably doesn't hurt that much, but on a hige paginaged list it might...?
  10. Benni Mack uploaded patch set 5 V +1 08/29/25, 14:08

  11. Benni Mack commented on patch set 5 08/29/25, 14:10

    DatabaseRecordList.php:654 Reply

    huge paginated list = ? - if we have 200 records, we fetch 200 full DB rows now. do you think this is too much? Record API only works with full records, if we have serious concerns on this topic, we shouldn't switch to Record API. Please elaborate.
  12. core-ci voted on patch set 5 V -1 08/29/25, 14:21 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/83995

  13. Garvin Hicking commented on patch set 5 08/29/25, 15:06

    DatabaseRecordList.php:654 Reply

    So currently let's say I have a recordlist that paginages a table "tx_geo_shapes". The table has columns: uid, pid, title, geojson plus various text/mm-relations (ecological data, animal populations, audience, state, postcode ...). The table has 10.000 records of every park in germany (this is a real-life project example). The shapes are stored in the DB instead of JSON files because Geospation operations are performed, and a mapserver is powered by the same table. The largest park has 50MB of shapes, the average is 5MB. Total data is 50GB. Currently, a recordlist will only fetch uid, pid and title for the list. TCA does not specify the JSON data, it is populated from outside imports (type=passthrough). Fetching 100 records and paginate through it until item 10.000 to also order/ filter by selected columns with metadata, and also export by CSV. Now if I understood this patch (sorry if I'm mistaken) now not only 10.000 x 255 bytes or so would be fetched and stored in PHP variables and transferred across the network (if DB is not local!), but 100x5MB=500 MB. And that column would not even be evaluated. Since the recordlist here is in a sysfolder with all other MM tables. displaying the recorda there would have a considerable performance / resource impact, right?
  14. Benni Mack commented on patch set 5 08/29/25, 15:30

    DatabaseRecordList.php:654 Reply

    OK. Thanks for the example. - MM relations are not evaluated directly (lazy loading) and only used when the column is shown. - If you show 100 parks at once and each row has 5MB in their geojson field, then yes, you will fetch 100*5MB through the network, even if the record is passthrough. So, once we work with larger records we run into this problem already. Extbase also does '*' and if you do a listing of 100 parks in the frontend, I assume you run into the same problem that you have a large database. Either we a) should stick with arrays here for list module b) come up with a "PartialRecord" concept that allows to only include selective fields c) we can keep the DB query as before and soften the concept of the existing Record API to also work with "partial" DB rows.
  15. Benni Mack uploaded patch set 6 V -1 08/29/25, 15:40

  16. core-ci voted on patch set 6 V -1 08/29/25, 15:54 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84000

  17. Garvin Hicking commented on patch set 6 08/29/25, 16:22

    DatabaseRecordList.php:654 Reply

    In Extbase/Frontend, I can work around this (custom queries, custom repositories). But in the Backend I couldn't. Maybe we could add an event that allows to modify "*" then, but I find it hacky. I would rather evaluate all registered TCA columns that are not pass-through (or unlisted). Then it could be "record api complete", as the extra rows would not contain so large data sets? Maybe with an event still we could filter type=json out as well....?
  18. Benni Mack uploaded patch set 7 V -1 08/29/25, 16:42

  19. Benni Mack uploaded patch set 8 08/29/25, 16:53

  20. Benni Mack commented on patch set 8 08/29/25, 16:56

    DatabaseRecordList.php:654 Reply

    See my code adaptions, works now.
  21. core-ci voted on patch set 7 V -1 08/29/25, 17:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84002

  22. core-ci voted on patch set 8 V -1 08/29/25, 17:09 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84007

  23. Garvin Hicking commented on patch set 8 08/29/25, 18:16

    DatabaseRecordList.php:654 Reply

    Thanks so much. Looking good, I will try to find time to test/review this properly (also post-merge, if you can get it in sooner, not blocking).

    Breaking-107356-UseRecordAPIInListModule.rst:21

    "expect" or "require" instead of "accept" maybe? Otherwise people may think it's "array|RecordInterface"?l (yes, the "instead" already expresses that, but I'd prefer to be explicit here)

    Breaking-107356-UseRecordAPIInListModule.rst:45

    Shall we use the "xclass" buzzword here for a better "search experience" of developers?
  1. Benni Mack uploaded patch set 9 V -1 09/01/25, 13:37

  2. core-ci voted on patch set 9 V +1 09/01/25, 13:44 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84159

  3. Benni Mack commented on patch set 9 09/01/25, 13:50

    Breaking-107356-UseRecordAPIInListModule.rst:21 Reply

    Done

    Breaking-107356-UseRecordAPIInListModule.rst:45 Reply

    Done
  4. Benni Mack updated the commit message in patch set 10 V +1 09/01/25, 14:53

  5. core-ci voted on patch set 10 V +1 09/01/25, 15:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84166

  6. Georg Ringer commented on patch set 10 V -1 09/01/25, 15:13

    getting error now: Table "fe_groups" has capability "InternalDescription" set but the record misses the corresponding field "description"
  7. Benni Mack uploaded patch set 11 V +1 09/01/25, 15:19

  8. core-ci voted on patch set 11 V +1 09/01/25, 15:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84172

  9. Georg Ringer voted on patch set 11 CR +1 V +1 09/01/25, 15:38

  10. Oli Bartsch commented on patch set 11 V -1 09/02/25, 08:57

    MethodCallMatcher.php:6442

    Adding those methods here does not work as they are reported all the time, even if called with the correct arguments.
  11. Benni Mack uploaded patch set 12 CR +1 V +1 09/02/25, 09:24

  12. Benni Mack commented on patch set 12 09/02/25, 09:24

    MethodCallMatcher.php:6442 Reply

    Done
  13. core-ci voted on patch set 12 V +1 09/02/25, 09:32 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84225

  14. Oli Bartsch uploaded patch set 13 V +1 09/02/25, 10:08

  15. Oli Bartsch commented on patch set 13 09/02/25, 10:11

    BackendUtility.php:2982

    Why not use capabilities here?

    BackendUtility.php:2989

    This field might be excluded, right?

    BackendUtility.php:2992

    SortBy is added below in L:3002

    BackendUtility.php:2995

    This field might be excluded, right?
  16. core-ci voted on patch set 13 V +1 09/02/25, 10:16 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84230

  17. Oli Bartsch commented on patch set 13 09/02/25, 10:39

    DatabaseRecordList.php:598

    What about DiffSourceField?

    DatabaseRecordList.php:806

    Can we just use $record->getLanguageId() here?

    DatabaseRecordList.php:1154

    Since we change the method signature, should we not directly use `Record $record` instead of `array $row`?

    DatabaseRecordList.php:1267

    IMO a real benefit would be to get rid of getProcessedValueExtra() here since we have the resolved record value already. I assume we however still have some distinctions here?

    DatabaseRecordList.php:1630

    This could also be `$record->getLanguageId()`?

    DatabaseRecordList.php:2096

    `$record->getLanguageInfo()` should be used?

    DatabaseRecordList.php:2201

    I think we should go with `RecordInterface $record` here. There is only one call passing an array, which we should adjust accordingly, reducing those `getRawRecord()->toArray()` calls as much as possible.

    DatabaseRecordList.php:2307

    We should work with the $record directly in this method.

    DatabaseRecordList.php:3139

    $record->getLanguageInfo()

    DatabaseRecordList.php:3140

    $record->getLanguageId()
  18. Oli Bartsch commented on patch set 13 09/02/25, 11:05

    DatabaseRecordList.php:598 Reply

    Done

    DatabaseRecordList.php:806 Reply

    No, as the interface does not cover this. But we can also not just call get() since this could trigger exceptions on missing fields.

    DatabaseRecordList.php:1580

    This needs to be RecordInterface
  19. Oli Bartsch uploaded patch set 14 V +1 09/02/25, 13:21

  20. Oli Bartsch commented on patch set 14 09/02/25, 13:21

    DatabaseRecordList.php:1154 Reply

    Done

    DatabaseRecordList.php:1267 Reply

    Not to be done in this patch i guess

    DatabaseRecordList.php:1580 Reply

    Done

    DatabaseRecordList.php:1630 Reply

    Done

    DatabaseRecordList.php:2096 Reply

    Done

    DatabaseRecordList.php:2201 Reply

    Done

    DatabaseRecordList.php:2307 Reply

    Done

    DatabaseRecordList.php:3139 Reply

    Done

    DatabaseRecordList.php:3140 Reply

    Done

    BackendUtility.php:2982 Reply

    Done

    BackendUtility.php:2989 Reply

    Done

    BackendUtility.php:2992 Reply

    Done

    BackendUtility.php:2995 Reply

    Done
  21. core-ci voted on patch set 14 V +1 09/02/25, 13:28 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84251

  22. Oli Bartsch voted on patch set 14 CR +1 V +1 09/02/25, 13:29

  23. Stefan Bürk voted on patch set 14 CR +1 V +1 09/02/25, 13:49

  24. Benni Mack commented on patch set 14 CR +2 V +2 09/02/25, 13:50

    Thanks for the work here.
  25. Benni Mack merged patch set 15 09/02/25, 13:50