77985[BUGFIX] Respect TCA field foreign_default_sortby by extbase
Commit message
Review
-
Code review CR +2 Done
- Benni Mack +2
- Alexander Schnitzler +1
-
Verified V +2 Done
- Benni Mack +2
- Alexander Schnitzler +1
- core-ci +1 CI
Issues 1
Files 7
- typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/ColumnMap.php6+17 −0
- typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapFactory.php9+2 −1
- typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapper.php4+42 −2
- typo3/sysext/extbase/Tests/Functional/Fixtures/Extensions/blog_example/Configuration/TCA/tx_blogexample_domain_model_post.php+1 −0
- typo3/sysext/extbase/Tests/Functional/Persistence/Fixtures/comments.csvnew+7 −0
- typo3/sysext/extbase/Tests/Functional/Persistence/Generic/Mapper/DataMapperTest.php+25 −0
- typo3/sysext/extbase/Tests/Unit/Persistence/Generic/Mapper/DataMapperTest.php5new+249 −0
Review log 5469
Show 3144 earlier messages
-
JAKOTA Design Group GmbH uploaded patch set 1 02/28/23, 22:17
-
core-ci voted on patch set 1 V -1 02/28/23, 23:19 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38235
-
JAKOTA Design Group GmbH uploaded patch set 2
V -102/28/23, 23:36 -
core-ci voted on patch set 2 V -1 03/01/23, 00:37 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38236
-
JAKOTA Design Group GmbH uploaded patch set 3
V -103/01/23, 00:51 -
core-ci voted on patch set 3 V +1 03/01/23, 01:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38237
-
Mathias Brodala commented on patch set 3 03/01/23, 07:39
No features for maintenance releases. But one could argue that this is actually a bug in Extbase. ;-)DataMapper.php:435You should use the QueryHelper::parseOrderBy() method instead here. -
JAKOTA Design Group GmbH uploaded patch set 4
V +103/01/23, 08:12 -
core-ci voted on patch set 4 V +1 03/01/23, 08:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38248
-
JAKOTA Design Group GmbH commented on patch set 4 03/01/23, 08:27
Reply
I would definitely categorize this as a bug in exbase. But I took the old patch series dating back to 2013 and updated it as good as I can and for version 6.0 it was a missing feature. ;) I would like some guidance here on how to proceed.DataMapper.php:435ReplyDone -
JAKOTA Design Group GmbH uploaded patch set 5
V +103/01/23, 08:35 -
core-ci voted on patch set 5 V +1 03/01/23, 08:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38250
-
Alexander Schnitzler commented on patch set 5 03/03/23, 08:33
Reply
Well, there are a lot of TCA features missing in extbase. We can call them bugs in extbase if you like. But I think people don't expect the order of records in the frontend to change after a patch release update, therefore I'd target version 12 exclusively. It's different with things that always worked way A, then broke and we need to fix. This always worked way B and now it works correctly. That's great but unexpected. To proceed further, we need tests here. Functional Tests to test the actual ordering result, a unit test maybe for the DataMapFactory that checks that the TCA is properly applied.ColumnMap.php:91Please use type declarations here and omit the @var annotation. -
JAKOTA Design Group GmbH commented on patch set 5 03/03/23, 12:23
Reply
The thing is though that the order is respected in the backend view and as an editor or developer I would expect the order in the frontend to match the order in the backend. Which means that everyone who is using this must have implemented a sorting of their own somewhere. I've seen this being done a lot in fluid or via viewhelpers. Like: https://viewhelpers.fluidtypo3.org/fluidtypo3/vhs/5.0.1/Iterator/Sort.html For all those folks this shouldn't break anything as they are sorting the data anyway. But I guess you're right. 😞 I thought about testing as well and would like to move the function getOrderingsForColumnMap from DataMapper.php to ColumnMap.php. This way it would be very easy to test this like all the other test. -
Alexander Schnitzler commented on patch set 5 03/03/23, 12:47
Reply
> The thing is though that the order is respected in the backend view and as an editor or developer I would expect the order in the frontend to match the order in the backend. I am not the one to decide whether this is considered breaking. So, let's start with calling this a bugfix and then see which versions receive it when this patch is done. > I thought about testing as well and would like to move the function getOrderingsForColumnMap from DataMapper.php to ColumnMap.php. This way it would be very easy to test this like all the other test. That's not a good idea because ColumnMap (and DataMap) are just data objects representing TCA configuration. They shouldn't contain any logic. DataMapper is the right class to fix this. -
JAKOTA Design Group GmbH uploaded patch set 6
V +103/03/23, 13:35 -
JAKOTA Design Group GmbH commented on patch set 6 03/03/23, 13:46
Reply
> I am not the one to decide whether this is considered breaking. So, let's start with calling this a bugfix and then see which versions receive it when this patch is done. > Sure, once we have a decision on that I'll change the commit message. > That's not a good idea because ColumnMap (and DataMap) are just data objects representing TCA configuration. They shouldn't contain any logic. DataMapper is the right class to fix this. Okay, I'll try to think of something. Any suggestions are welcome.ColumnMap.php:91ReplyDone -
Alexander Schnitzler commented on patch set 6 03/03/23, 14:03
Reply
I just had a look at where the `getChildSortByFieldName()` is used and it's not just in the DataMapper but also in the Backend class. Changing things like sorting in those classes is a bit more complex than other small bug fixes, so I'd definitely go for functional tests here. Do you know how to write those? Because, to be honest, if you don't yet, this will be hard to get into a mergeable state where others are confident enough to give a positive vote. -
core-ci voted on patch set 6 V -1 03/03/23, 14:07 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38360
-
JAKOTA Design Group GmbH uploaded patch set 7
V -103/03/23, 14:13 -
core-ci voted on patch set 7 V +1 03/03/23, 14:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38362
-
JAKOTA Design Group GmbH commented on patch set 7 03/03/23, 14:39
Reply
Nope, I have no clue how to do that yet. But I believe a unit test should suffice. The way this fix works in contrast to the former once is that the data for "getChildSortByFieldName" is not changed, as those are also used to do the sorting on writing to the DB. That is where the field foreign_default_sortby comes in. Which is only used for sorting when reading data. But if it turns out that a functional test is a hard requirement, I'll have to figure it out. It can't be that hard. (Famous last words) -
JAKOTA Design Group GmbH uploaded patch set 8 03/04/23, 14:38
Copied Votes: * Verified+1 (copy condition: "changekind:NO_CHANGE") -
JAKOTA Design Group GmbH uploaded patch set 9
V +103/04/23, 14:38 -
core-ci voted on patch set 8 V +1 03/04/23, 15:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38443
-
core-ci voted on patch set 9 V -1 03/04/23, 15:35 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38444
-
Alexander Schnitzler commented on patch set 9 03/04/23, 16:06
DataMapperTest.php:26Well, thanks for your effort here and I don't like to be the one to complain but I have to say I dislike the unit test approach where the unit under test is mocked, even just partially. Those tests (there are plenty in extbase) are a pain in the ass if you want to refactor class internals. Luckily, both methods under test fit the rough rules I do apply to unit tests. Small, side effect free methods or method chains that don't need a mocked dependency. My suggestion would be to make `getOrderingsForColumnMap` public and test it without a mocked datamapper. If I had to decide, I would also make `setOneToOneRelation` public. Very often, we do think that public methods are our api, but they are not. The underlying interface is. So, as long as the contract is used to query functionality, I don't see a problem with people starting to call those methods out of context. Anyway, that's just my personal opinion based on experience on that field. If we want to have tests and it can't be functional tests, then I'd rather make methods public than mocking the test subject. -
JAKOTA Design Group GmbH uploaded patch set 10
V -103/05/23, 10:21 -
JAKOTA Design Group GmbH commented on patch set 10 03/05/23, 10:44
DataMapperTest.php:26ReplyDon't worry about it, complain as much as you need. I'm all for productive criticism. I usually would just find a bug, develop a fix, dump it to the issue system and let someone else deal with the rest. But I'm kind of invested in this one. So your help is greatly appreciated. I've made the functions public and moved the null checks so the return of getOrderingsForColumnMap always has the expected values. As for the tests, I rewrote them and took some inspiration from other unit test. And I hope I understood you right. Last thought: As the DataMapper class already relies on DataMapFactory and DataMapFactory has some logic to deal with data structure would'd it make sene to move getOrderingsForColumnMap to DataMapFactory? This way the testing could be done in the already existing DataMapFactoryTest. -
core-ci voted on patch set 10 V -1 03/05/23, 11:36 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38484
-
Alexander Schnitzler uploaded patch set 11
V -103/05/23, 12:07 -
Alexander Schnitzler commented on patch set 11 03/05/23, 12:11
DataMapperTest.php:26ReplyHey, I took the liberty to do the following things: - Rearranged the unit tests to visually separate arrangement, acting and assertion. IMHO those tests are way easier to read this way. - I added two more tests to cover edge cases. - I changed getOrderingsForColumnMap to return null instead of an empty array (also covered with a test) - I added a functional test to check if the TCA config is actually applied. -
Alexander Schnitzler commented on patch set 11 03/05/23, 12:22
DataMapFactory.php:357This is actually not needed because the 1:1 relation always fetches a specific object and the order is irrelevant. I'd suggest reverting this change and the unit tests for this case.DataMapper.php:429This actually doesn't work. We are dealing with an MM table here and extbase does not respect the `foreign_default_sortby` with this change because `ColumnMap::$childDefaultSortByFieldName` is never set in `\TYPO3\CMS\Extbase\Persistence\Generic\Mapper\DataMapFactory::setManyToManyRelation()`. However, I don't even think the core itself supports MM tables along with `foreign_default_sortby`, probably not even `foreign_sortby`, except for group fields maybe, which store the relation as lists on the parent. But this needs to be checked first. -
Alexander Schnitzler voted on patch set 11 V +1 CR -1 03/05/23, 12:22
-
JAKOTA Design Group GmbH uploaded patch set 12
CR -1V +103/05/23, 12:49 -
core-ci voted on patch set 11 V -1 03/05/23, 12:49 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38489
-
JAKOTA Design Group GmbH commented on patch set 12 03/05/23, 12:53
DataMapFactory.php:357ReplyDoneDataMapper.php:429ReplyAckDataMapperTest.php:26ReplyThank you very much. -
JAKOTA Design Group GmbH commented on patch set 12 03/05/23, 13:03
DataMapFactory.php:357ReplySorry, I edited the wrong part... -
Alexander Schnitzler commented on patch set 12 03/05/23, 13:04
DataMapperTest.php:26ReplyYou're welcome. -
JAKOTA Design Group GmbH uploaded patch set 13 03/05/23, 13:13
-
JAKOTA Design Group GmbH commented on patch set 13 03/05/23, 13:14
DataMapFactory.php:357ReplyDone -
core-ci voted on patch set 12 V -1 03/05/23, 13:17 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38491
-
core-ci voted on patch set 13 V +1 03/05/23, 13:19 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38492
-
Alexander Schnitzler commented on patch set 13 03/05/23, 13:29
Alright, there is one last thing that needs to be tested. The DataMapperTest only checks, that the DataMapper respects the ordering field but we don't know yet how this interacts with the default sorting of repositories. My assumption would be that foreign_default_sortby only comes into play with queries that don't have a sorting set yet. Would you agree on this?
-
Alexander Schnitzler commented on patch set 13 03/05/23, 13:30
Reply
Done -
JAKOTA Design Group GmbH uploaded patch set 14
V +103/05/23, 13:32 -
core-ci voted on patch set 14 V +1 03/05/23, 13:39 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38494
-
Alexander Schnitzler commented on patch set 14 03/05/23, 13:43
Reply
To be more precise, let's use the example of Blog posts to comments. ``` $query->setOrderings([ 'comments.date' => QueryInterface::ORDER_DESCENDING, ]); ``` This should be possible in extbase. And in this case, I would assume that `foreign_default_sortby = uid desc` does not overwrite the default ordering of the repository. -
Alexander Schnitzler commented on patch set 14 03/05/23, 13:45
Reply
Neither should `default_sortby` and if that's the case, we already have another bug. So maybe, this is something for another patch. I still would like to check the current behaviour to know where this path leads to. -
Alexander Schnitzler voted on patch set 14 V +1 CR +1 03/05/23, 13:46
-
JAKOTA Design Group GmbH commented on patch set 14 03/05/23, 13:50
Reply
Yes I would, but if I understand the code right, as it is before this patch, than the sorting of repositories if always overridden as soon as getChildSortByFieldName != null. So I'm not sure what takes precedence here. -
Alexander Schnitzler commented on patch set 14 03/05/23, 13:54
Reply
I will step debug this and consider this another case. I'll mark this resolved. Let's wait for more reviews of others. IMO this can be merged. You just need to think about FEATURE vs. BUGFIX again ;) -
Alexander Schnitzler commented on patch set 14 CR -1 03/05/23, 13:54
-1 because of open question of FEATURE vs. BUGFIX. -
JAKOTA Design Group GmbH commented on patch set 14 03/05/23, 14:24
Reply
Thanks, I'll leave it like this for a while. Hoping others will have opinions on this. But I tend to changing it to Bugfix. -
Benni Mack commented on patch set 14 03/06/23, 08:54
this is a bugfix -
JAKOTA Design Group GmbH commented on patch set 14 03/06/23, 12:34
ColumnMap.php:85This might be an copy paste error on my part. But I think this should be cleared up. This value can hold multiple fields for sorting not just on. So it should say: "The name of the fields the results from the child's table are sorted by default" And rename childDefaultSortByFieldName to childDefaultSortByFields like relationTableMatchFields except it is a comma separated string not an array Plus renaming all the function and so on. Any thought on that? -
Alexander Schnitzler commented on patch set 14 CR 0 03/06/23, 13:20
ColumnMap.php:85ReplyI would name it `defaultChildTableSortings` because the TCA does not only hold fields but also the sorting order. Then, I would store the value as string because the initial idea of a column map was to replicate TCA information in object form, without parsing the data. This can be done in the future with refactorings. Just my `IMHO` here. -
JAKOTA Design Group GmbH commented on patch set 14 03/06/23, 14:32
ColumnMap.php:85ReplyThat kind of ruins the naming scheme, a bit. All the other properties start with parent or child, except when they don't.... How about childTableDefaultSortings? Which would also fit nicely to childTableName. 1+ to for keeping the value as string. -
JAKOTA Design Group GmbH uploaded patch set 15
V +103/07/23, 09:46 -
JAKOTA Design Group GmbH commented on patch set 15 03/07/23, 09:49
ColumnMap.php:85ReplyDone -
core-ci voted on patch set 15 V +1 03/07/23, 09:53 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/38586
-
Alexander Schnitzler voted on patch set 15 V +1 CR +1 03/07/23, 11:44
-
Benni Mack voted on patch set 15 V +2 CR +2 03/07/23, 11:48
-
Benni Mack merged patch set 16 03/07/23, 11:48
-
Benni Mack commented on patch set 16 03/08/23, 08:47
DataMapFactory.php:372I guess we should have kept this protected. The main reason to have this public is to do unit testing easier. -
Alexander Schnitzler commented on patch set 16 03/08/23, 08:54
DataMapFactory.php:372ReplyWe can easily change this back but I would still argue against "`public` means api". This method, like many others too, is side effect free. No change of internal state. I'd like to challenge the idea that everything needs to be protected/hidden which is not called by the framework itself from the outside. Think about this: If someone comes along, makes this a service, we all suddenly agree on it being public. ;) -
Oliver Klee commented on patch set 16 03/08/23, 09:25
DataMapFactory.php:372ReplyI understand "public means API" as "if it's public, people will start using it", and we'll have a harder time refactoring the method by changing its name, its contract or by removing it. -
Benni Mack commented on patch set 16 03/08/23, 09:48
DataMapFactory.php:372ReplyOverall, we can resolve this topic for now, as the whole class is marked as "not part of the API" :) -
Alexander Schnitzler commented on patch set 16 03/08/23, 10:29
DataMapFactory.php:372ReplyMaybe we need to make more use of `@internal`. With the switch from explicitly api via `@api` to explicitly non-api via `@internal` a lot of methods are considered api that aren't. Even many that always have been public. I am always happy if a class is marked `@internal` if I work on something. ^^