Review main Abandoned

67172[BUGFIX] Change MM handling with l10n_mode exclude

David Hedden CR 0 V 0 Normal +72 −39 updated 03/06/26, 16:52

Commit message

Previously, MM relations with `l10n_mode=>"exclude"` were handled by
synchronizing the `uid_local` and `uid_foreign` of all the translated
records. This was done each time any of the records were modified.

One problem with this approach was that only the translations
of the current record were handled while saving the relation.
The `uid_local` was saved for default + all translations,
but `uid_foreign` was always the `uid` of the record in the default language.
All other relations using any of the used `uid_local`'s were deleted.
The reverse of this was also true, so if the record with `uid_foreign`
is the one being saved. That meant when showing one of these translated records,
it would have missing relations, if the counterpart was saved last.
Also, if language fallback was active, it could show duplicated entries
for each translation of the relation.

The new approach is to only save the relation of the records
in the default language and let extbase handle the overlaying.
This is done by using the `uid` instead of the `_localizedUid` while
building the constraints for a translated record.
For non-extbase fetching of the relations, the `l10n_parent` can be used.

Review

  • Code review CR 0
  • Verified V 0

Issues 3

Files 9

Review log 5782

Show 3757 earlier messages
  1. David Hedden uploaded patch set 1 12/17/20, 09:22

  2. TYPO3com voted on patch set 1 V -1 12/17/20, 09:23 Completed build in 20s on Thu, 17 Dec, 09:23 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-8221

  3. Georg Ringer commented on patch set 1 12/17/20, 13:33

    IMO -1 should be allowed as well
  4. David Hedden commented on patch set 1 12/17/20, 14:12

    Reply

    I think it should be fine, since the condition in DataMapProcessor is only to avoid the equals function to use the localized_uid of the record. And that shouldn't be set for -1 records ?!
  5. David Hedden uploaded patch set 2 12/17/20, 14:42

  6. TYPO3com voted on patch set 2 V +1 12/17/20, 14:52 Completed build in 09m 44s on Thu, 17 Dec, 02:52 PM Test Summary: 32496 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-8230

  7. David Hedden commented on patch set 2 12/17/20, 16:12

    Reply

    Correction, i meant the condition in DataMapper
  8. Steffen Frese voted on patch set 2 V +1 CR +1 06/15/21, 10:38

  9. Steffen Frese wrote 06/15/21, 10:38

    Removed Verified+1 by Steffen Frese <steffenf14@gmail.com>
  10. Steffen Frese voted on patch set 2 V +1 06/15/21, 10:48

  11. Benni Mack uploaded patch set 3 07/29/21, 15:32

  12. core-ci voted on patch set 3 V +1 07/29/21, 15:56 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/13888

  13. Benni Mack uploaded patch set 4 09/14/21, 18:45

  14. core-ci voted on patch set 4 V +1 09/14/21, 18:58 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/16717

  15. Benni Mack commented on patch set 4 CR +1 11/16/21, 20:28

    good change, I *do* think we need to have tests here though
  16. Benni Mack updated the commit message in patch set 5 11/18/21, 10:47

  17. Benni Mack commented on patch set 5 CR -1 11/18/21, 11:46

    We need (CSV / DataHandler) tests for Workspaces + l10n_mode exclude + MM
  18. core-ci voted on patch set 5 V +1 11/18/21, 11:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/19474

  19. Benni Mack wrote 11/29/21, 10:19

    Change destination moved from master to main
  20. Benni Mack updated the commit message in patch set 6 11/29/21, 14:06

  21. core-ci voted on patch set 6 V +1 11/29/21, 16:44 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/20091

  22. Benni Mack updated the commit message in patch set 7 V +1 05/03/23, 09:04

  23. core-ci voted on patch set 7 V -1 05/03/23, 09:07 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41377

  24. Benni Mack uploaded patch set 8 V -1 07/06/23, 11:50

  25. core-ci voted on patch set 8 V +1 07/06/23, 12:12 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43712

  26. Benni Mack commented on patch set 9 V +1 07/06/23, 18:29

    Published edit on patch set 8.
  27. core-ci voted on patch set 9 V +1 07/06/23, 18:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43805

  28. Christian Kuhn commented on patch set 9 07/07/23, 09:37

    should we mark this [wip] until the patch has test coverage?
  29. Benni Mack marked it work in progress 07/07/23, 11:41

  30. David Hedden uploaded patch set 10 V +1 03/13/24, 12:45

  31. core-ci voted on patch set 10 V -1 03/13/24, 12:52 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54476

  32. David Hedden uploaded patch set 11 V -1 03/13/24, 21:56

  33. core-ci voted on patch set 11 V -1 03/13/24, 22:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54503

  34. David Hedden uploaded patch set 12 V -1 03/13/24, 22:18

  35. core-ci voted on patch set 12 V -1 03/13/24, 22:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54504

  36. David Hedden uploaded patch set 13 V -1 03/13/24, 22:31

  37. core-ci voted on patch set 13 V +1 03/13/24, 22:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54505

  38. David Hedden uploaded patch set 14 V +1 03/14/24, 17:23

  39. core-ci voted on patch set 14 V -1 03/14/24, 17:30 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54548

  40. David Hedden uploaded patch set 15 V -1 03/14/24, 18:15

  41. core-ci voted on patch set 15 V +1 03/14/24, 18:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54552

  42. David Hedden uploaded patch set 16 V +1 03/15/24, 16:07

  43. core-ci voted on patch set 16 V +1 03/15/24, 16:12 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54600

  44. David Hedden marked it ready for review 03/16/24, 11:06

  45. Benni Mack commented on patch set 16 03/16/24, 11:28

    Hey David, awesome improvement. With these kind of changes, I feel we need a bullet-proof commit message so our future-selves really know this change. Can you add in what situations this change now makes a difference, how it was before, and what issues are now solved (since we are now setting a new "this is the way to go" default which is properly tested)
  46. David Hedden updated the commit message in patch set 17 V +1 03/16/24, 16:59

  47. core-ci voted on patch set 17 V +1 03/16/24, 17:05 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54628

  48. David Hedden marked it work in progress 03/16/24, 18:39

  49. David Hedden commented on patch set 17 03/16/24, 18:52

    Reply

    I've changed the commit msg, i hope this is sufficient. While writing another test to cover the extbase fallback logic, i might have found a problem with this change related to workspaces, so i marked it as WIP till i have it sorted.
  50. David Hedden uploaded patch set 18 V +1 03/17/24, 11:42

  51. core-ci voted on patch set 18 V +1 03/17/24, 11:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54637

  52. David Hedden updated the commit message in patch set 19 V +1 03/17/24, 12:02

  53. core-ci voted on patch set 19 V +1 03/17/24, 12:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54638

  54. David Hedden commented on patch set 19 03/17/24, 12:15

    This change is ready for review.

    DataMapper.php:553

    While writing the extbase test this was throwing an error duo $value not being a Object anymore. EZ to avoid by testing for the correct instance. But i tested workspaces handling for MM relations and ran all the test without this block an everything worked fine. Also with a breakpoint at the place were the value was set i could not find a scenario were it was hit. As i understand it everything should be handled by $this->resolveRelationValuesOfField. Can somebody verify that it's not needed anymore or tell me a scenario were this would be needed? Than i could test with this behavior in mind.
  55. David Hedden updated the commit message in patch set 20 V +1 03/17/24, 12:23

  56. David Hedden updated the commit message in patch set 21 03/17/24, 12:24

  57. core-ci voted on patch set 20 V +1 03/17/24, 12:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54639

  1. core-ci voted on patch set 21 V +1 03/17/24, 12:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54640

  2. Rémy DANIEL commented on patch set 21 03/17/24, 23:26

    DataMapper.php:553 Reply

    Hi. I have a pending review on this piece of code. Maybe the commit message can help you understanding its purpose.
  3. Rémy DANIEL commented on patch set 21 03/17/24, 23:26

    DataMapper.php:553 Reply

  4. David Hedden commented on patch set 21 03/18/24, 08:55

    DataMapper.php:553 Reply

    The changes you have linked are not in the exact part I'm referring to. They are in the first if condition when $resolvedRelationIds is not empty. I'm asking myself in what case could this variable be empty while there is still a mm relation in workspace to handle.

    DataMapper.php:553 Reply Unresolved

    Your change is not affecting the part im refering to, all the changes are if
  5. Benni Mack commented on patch set 21 03/23/24, 20:58

    TranslationTest.php:435

    what happens if we do not use Overlays = free mode? This is a supported use-case in Frontend (not just Extbase).
  6. David Hedden uploaded patch set 22 V +1 03/24/24, 09:07

  7. core-ci voted on patch set 22 V +1 03/24/24, 09:12 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54900

  8. David Hedden commented on patch set 22 03/24/24, 09:36

    TranslationTest.php:435 Reply

    I changed the test to use `overlays_off`. With `overlays_on` it should've failed, since in the test data tag's are not translated. The problem is that not only the tags with l10n_mode=exclude are in the post, also the non-translated blog is provided. Should this be then handled as another issue?
  9. David Hedden uploaded patch set 23 V +1 03/24/24, 13:42

  10. core-ci voted on patch set 23 V -1 03/24/24, 13:48 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54901

  11. David Hedden commented on patch set 23 03/24/24, 14:14

    TranslationTest.php:435 Reply

    Okay, it shouldn't work with `OVERLAYS_OFF` either. Since the documentation states: "off" - do not do overlay, only fetch records available in the current "contentId". As is understand that, it would mean that if we query a language that is not from the current contentId it should return northing. In this case it's a sub-record but it should still apply. So the only valid mode would be mixed. This seems like a general issue with fetching relations, since `OVERLAYS_OFF` and `OVERLAYS_ON` both should not fetch any non-translated records. But in both cases tag & blog are attached to the post, even though they are not translated. I changed the tests to cover at least the case for `OVERLAYS_ON` when a translated tag is present. And for `OVERLAYS_MIXED` if there is a non translated tag used.
  12. David Hedden uploaded patch set 24 V -1 03/24/24, 14:24

  13. core-ci voted on patch set 24 V +1 03/24/24, 14:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54902

  14. David Hedden uploaded patch set 25 V +1 03/24/24, 14:41

  15. core-ci voted on patch set 25 V +1 03/24/24, 14:47 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/54903

  16. David Hedden commented on patch set 25 03/25/24, 20:30

    TranslationTest.php:435 Reply

    So after checking again, this behavior is intended atm. The confusion came because the system this issue came up for me has this "patched", so default language will be discarded in those cases. https://forge.typo3.org/issues/88137
  17. Astrid Haubold commented on patch set 25 V -1 04/05/24, 11:48

    We discussed this in the translation handling initiative today. While we see that this approach makes sense and would solve some long standing issues, we think it might lead into a direction that is not desirable on the long run: our original idea was to be able to abandon overlaying completely at some point and always use records of target language directly instead. This patch would on the contrary introduce more overlaying. We would like to hold the patch for now to leave time for further discussion.
  18. Jo Hasenau commented on patch set 25 CR -1 V -1 04/05/24, 12:53

    Reply

    Since this was not actually resolved, I just removed the checkbox :-)
  19. David Hedden commented on patch set 25 04/08/24, 10:24

    Reply Unresolved

    To clarify a bit, this change removes one layer of overlaying. If the intent is to remove the reliance on the base language and overlaying in the future, it would be possible to directly fetch the translated record via l10n_parent. But this would require some breaking changes in the core behavior first. ``` // Typo3DbBackend // We have a translated record from the DB, but we do overlays, so let's take the default language record // and do overlays again later-on ``` Before: relation -> fetch translated record (DataMapper) -> fetch base record (Typo3DbBackend) -> overlay language (PageRepository) Patch: relation -> fetch base record (DataMapper) -> overlay language (PageRepository)
  20. David Hedden commented on patch set 25 05/17/24, 11:05

    If there is still interest in this, there is a general issue that is affecting this change. It is related to the DataMapper that is not taking the current LanguageAspect in to account when saving the object to the runtime cache.
  21. Garvin Hicking commented on patch set 25 08/02/24, 09:44

    Here's a ticket update maybe worth adressing: https://forge.typo3.org/issues/92777#note-14
  22. Benni Mack updated the commit message in patch set 26 CR -1 V +1 10/18/24, 13:41

  23. Benni Mack commented on patch set 26 02/15/26, 17:41

    Hey all, I created a draft to build a solution for our MM relation issues (https://forge.typo3.org/issues/108986). You can see there are a lot of them, and IMHO the only solution is to 1. always look from the local side 2. always add sys_language_uid and t3ver_wsid Are you cool with this approach? Please comment, I think we can pave the way until v14 LTS to build this foundation non-breaking (although I'd like to add the new DB fields before the first LTS release)
  24. Eric Harrer commented on patch set 26 02/20/26, 11:39

    Reply Unresolved

    Thank you for the promising idea. We think it's a good one from the perspective of the Translation Handling Initiative. We have discussed your approach and agree with it. Basically, we had language consistency on both sides (uid_local/uid_foreign) of the mm references in mind, but some preparation is still necessary before this will become possible. For this reason, we see your solution as a well-structured and uniform basis for possible future optimizations in the area of MM tables.
  25. David Hedden abandoned the change 03/06/26, 16:52