74450[TASK] Reduce sql queries for page link generation
Commit message
Review
-
Code review CR +2 Done
- Benni Mack +2
- Markus Klein +1
- Oliver Hader +1
-
Verified V +2 Done
- Benni Mack +2
- Markus Klein +1
- Oliver Hader +1
- core-ci +1 CI
Issues 1
Files 5
Review log 5378
Show 3453 earlier messages
-
Christoph Lehmann uploaded patch set 1 04/27/22, 22:15
-
core-ci voted on patch set 1 V -1 04/27/22, 22:25 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26540
-
Stefan Bürk commented on patch set 1 CR -1 04/27/22, 23:03
I read the issue, and the "proposed" hook implemention. The used hook fully overrides the $disableGroupAccessCheck, telling that to disable the group access check for all getPage() calls. This means, that if code calls with and without disabled group access check, all calls will retrieve the result as if the group access check was disabled. Fully ignoring any set group restrictions. Provided code however cannot adopt or do the same thing. with and withoud disabled group access checks may return different results, so reusing the result will introduce invalid result return for quite some installations. And the used constellation tries to reuse the result of the "group access not disabled" check, which may be an empty result for the "disabled group access" check - return wrong informations. And this generally. This is a hard breaking change. This cannot be done with second cache and a assumpation that the other cache variant contains the same result - as the assumptation is not correct in all constellations. Thus breaking core tests properly. Maybe it have to be looked up in the menu generation, if two calls can avoided some how. Doing it in the getPage() method is not the correct spot. Simulating or integrating the "example hook" as "FEATURE" would be more some kind of global configuration which is checked here and overrides it. But I'm not sure if this kind if feature configuration makes that much sense. It seems that this topic has been underestimated. Maybe describing the full use case more in the issue, so we can decide how to implement this in a more global way. Simply avoiding some sql calls may seem a optimization, but eleminating needed queries for correct working is not a goal. -
Stefan Bürk commented on patch set 1 04/27/22, 23:04
Reply
However thanks for the patch - but I think the approch is the wrong one. -
Christoph Lehmann updated the commit message in patch set 2 04/28/22, 11:11
-
Christoph Lehmann commented on patch set 2 04/28/22, 11:11
Reply
Thanks for your detailed reply! The example in the issue looks indeed like a new global setting for systems not using frontend group restrictions at all. >Simply avoiding some sql calls may seem a optimization, but eleminating needed queries for correct working is not a goal. What i want to change with this commit is reducing the sql queries used to generate a page link. Apparently it looked straight forward, that, when a page was 'authorized' then the result could be used a second time for for a queries where frontend group access check doesn't matter. But yes, tests break. Do you see a fault in this assumption generally? I will give it another try. -
core-ci voted on patch set 2 V -1 04/28/22, 11:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26578
-
Stefan Bürk commented on patch set 2 04/28/22, 16:53
Reply
Firstly we should investigate why there are two calls run, with and without check. Their must be a reason for it. And I'm not a fan of reusing a result of a completly other query. However, you cannot reuse the result of the "not disabled group check" if existing, as this may be "empty" or "null" as the check has given the result that the user has not the corresponding access. And thus, reusing this for the "disabled check" as result is wrong. It could be done the way arround - but depending on the issue, the not disable check is the first, and the other the second .. which will not change anything in "that" case. To summerize this - I would say it's the wrong approach and not really doable this way.Maybe the two calls are not communing from the menu itself. And generally does not hurt at all, as this is only for the menu rendering. And tasks which should be done should be done, without "faking" or "dirty workarounding" to avoid legitim queries. Generelly, if possible taking superflous or unneeded queries away is a nice attempt. Should be investigated if it can be ensured that the call is done at all to "getPage()" with and without group check. As the cache/result faking is a errounous approach at all. -
Stefan Bürk updated the commit message in patch set 3 04/28/22, 16:53
-
core-ci voted on patch set 3 V -1 04/28/22, 17:01 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26616
-
Christoph Lehmann uploaded patch set 4 04/28/22, 21:28
-
core-ci voted on patch set 4 V -1 04/28/22, 21:59 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26624
-
Christoph Lehmann updated the commit message in patch set 5 04/28/22, 22:08
-
Christoph Lehmann updated the commit message in patch set 6 04/28/22, 22:09
-
Christoph Lehmann uploaded patch set 7 04/28/22, 22:16
-
core-ci voted on patch set 5 V -1 04/28/22, 22:38 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26626
-
core-ci voted on patch set 6 V -1 04/28/22, 22:39 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26627
-
core-ci voted on patch set 7 V -1 04/28/22, 22:47 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26628
-
core-ci voted on patch set 7 V +1 04/28/22, 22:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26628
-
Christoph Lehmann uploaded patch set 8 04/29/22, 21:28
-
core-ci voted on patch set 8 V -1 04/29/22, 21:36 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26686
-
Christoph Lehmann updated the commit message in patch set 9 04/29/22, 21:37
-
Christoph Lehmann commented on patch set 9 04/29/22, 21:37
Reply
Done -
core-ci voted on patch set 8 V +1 04/29/22, 21:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26686
-
core-ci voted on patch set 9 V -1 04/29/22, 22:07 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26687
-
core-ci voted on patch set 9 V +1 04/29/22, 22:15 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26687
-
Markus Klein commented on patch set 9 04/29/22, 23:36
PageRouter.php:255(int)($route['sys_language_uid'] ?? -1) === $language->getLanguageId() If $route does not have the sys_language_uid (for whatever reason) avoid a PHP 8 warning. I chose -1 as value to ensure the condition fails. Please add this as a comment to clarify the magic number.PageLinkBuilder.php:496Are you sure $page always has all necessary fields? (I didn't check) -
Christoph Lehmann uploaded patch set 10 05/01/22, 17:45
-
Christoph Lehmann commented on patch set 10 05/01/22, 17:54
PageRouter.php:255ReplyAll calls to generateUri() in core pass a pageId, with the exception of the PageLinkBuilder. There, see build(), the page is fully resolved and overlayed. You are right, we can not know what 3rd party extensions pass as page argument. Thus i added the '_resolved' option to make sure $route is what we expect.PageLinkBuilder.php:496ReplyDone -
Christoph Lehmann commented on patch set 10 05/01/22, 17:55
PageRouter.php:255ReplyDone -
core-ci voted on patch set 10 V +1 05/01/22, 18:01 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26714
-
Oliver Hader commented on patch set 10 05/02/22, 07:56
I see the point of this patch (enhancing the regular use-case), however I have the feeling we probably need some more tests for language handling scenarios (in terms of "I'm wondering that no tests are failing here")PageRouter.php:259The information of `l10n_parent` seems to be lost here. Besides that, a bunch of more checks are required, e.g. whether that array actually has the required values, like `uid` or `sys_language_uid`... -
Oliver Hader voted on patch set 10 CR -1 05/02/22, 07:57
-
Christoph Lehmann uploaded patch set 11 05/07/22, 14:21
-
core-ci voted on patch set 11 V -1 05/07/22, 14:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26962
-
Stefan Bürk commented on patch set 11 05/07/22, 14:56
Rebase the patch upon actual main branch please. The test failure are related to wrong dind image used, which has been pinned with another patch yesterday. -
Christoph Lehmann uploaded patch set 12 05/07/22, 18:18
-
core-ci voted on patch set 12 V -1 05/07/22, 18:31 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26976
-
Christoph Lehmann uploaded patch set 13 05/07/22, 18:36
-
Christoph Lehmann commented on patch set 12 05/07/22, 18:41
Reply
No tests fail, because everything is well ;-) Given $resolvedOption is true, then $route is a product of PageRepository::resolvePage() and thus a result of PageRepository::getPage() See stacktrace until here: PageRouter.php:232, TYPO3\CMS\Core\Routing\PageRouter->generateUri() PageLinkBuilder.php:500, TYPO3\CMS\Frontend\Typolink\PageLinkBuilder->generateUrlForPageWithSiteConfiguration() PageLinkBuilder.php:162, TYPO3\CMS\Frontend\Typolink\PageLinkBuilder->build() There are a lot of page link tests, see SlugLinkGeneratorTest and TypoLinkGeneratorTest. Which case do you miss?PageRouter.php:259ReplyI added more basic checks. -
core-ci voted on patch set 13 V -1 05/07/22, 18:44 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26977
-
Christoph Lehmann commented on patch set 13 05/07/22, 18:46
> Patch Set 11: > > Rebase the patch upon actual main branch please. > > The test failure are related to wrong dind image used, which has been pinned with another patch yesterday. Thanks, i did. Now grunt seems not happy, see https://git.typo3.org/typo3/CI/cms/-/jobs/1274553 -
core-ci voted on patch set 13 V +1 05/07/22, 18:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26977
-
Stefan Bürk commented on patch set 13 05/07/22, 18:55
Retried the run. the yarn download seems to faile randomly since a few days ... it's on the todo for next week. Unrelated to the patch however. Howecer out until mo day evening .. so not possible for me to proper review/test/help with the patch until then. -
Christoph Lehmann updated the commit message in patch set 14 05/07/22, 19:18
-
Christoph Lehmann commented on patch set 14 05/07/22, 19:18
> Patch Set 13: > > Retried the run. the yarn download seems to faile randomly since a few days ... it's on the todo for next week. Unrelated to the patch however. > > Howecer out until mo day evening .. so not possible for me to proper review/test/help with the patch until then. Okay, thank you! -
core-ci voted on patch set 14 V +1 05/07/22, 19:26 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/26980
-
Markus Klein commented on patch set 14 05/09/22, 08:42
PageRouter.php:264No need to compare to "true". $route['_PAGES_OVERLAY'] ?? false is enough -
Christoph Lehmann commented on patch set 15 05/20/22, 07:56
Published edit on patch set 14. -
core-ci voted on patch set 15 V +1 05/20/22, 08:03 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/27343
-
Christoph Lehmann commented on patch set 15 05/23/22, 21:02
PageRouter.php:264ReplyDone -
Christoph Lehmann commented on patch set 15 10/20/22, 20:33
It reduces up to 50 percent of SQL queries for generating a single page link. Would love to see that it get merged. What a test needs to be added? -
Markus Klein commented on patch set 15 10/20/22, 22:10
PageRouter.php:260$resolvedOption is already bool, no need to compare it again
-
Markus Klein commented on patch set 16 10/20/22, 22:10
Published edit on patch set 15. -
Markus Klein voted on patch set 16 CR +1 10/20/22, 22:11
-
core-ci voted on patch set 16 V -1 10/20/22, 22:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/32538
-
Stefan Bürk rebased onto patch set 17 10/20/22, 22:54
-
core-ci voted on patch set 17 V +1 10/20/22, 23:02 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/32542
-
Christoph Lehmann commented on patch set 17 10/21/22, 07:39
PageRouter.php:260Replythank you! -
Benni Mack commented on patch set 17 12/07/22, 09:31
Found a good test case in v11.5. First hit had 1700 SQL queries without the patch, but 1200 SQL queries WITH the patch applied. -
Markus Klein commented on patch set 17 12/07/22, 09:46
Reply
😎 -
Anja Leichsenring commented on patch set 17 V +1 CR +1 12/07/22, 19:40
found not a so dramatic case as Benni, but could reproduce the reduction and did not run into any errors. Nice one, thanks. -
Benni Mack commented on patch set 17 CR -1 12/07/22, 22:25
will work on that. -
Benni Mack uploaded patch set 18
CR +1V +101/09/23, 22:39 -
Benni Mack uploaded patch set 19 01/09/23, 22:40
-
Benni Mack uploaded patch set 20 01/09/23, 22:41
-
core-ci voted on patch set 19 V +1 01/09/23, 22:52 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/35622
-
core-ci voted on patch set 20 V +1 01/09/23, 22:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/35623
-
core-ci voted on patch set 18 V -1 01/09/23, 23:10 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/35621
-
Markus Klein voted on patch set 20 CR +1 01/10/23, 08:03
-
Oliver Hader commented on patch set 20 CR +1 01/10/23, 08:25
PageRouter.php:245Keeping this "3rd party input" block for backward compatibility seems to be fine. In the long term (v12+) I think we should remove it to reduce complexity and dependency a bit... -
core-ci voted on patch set 18 V +1 01/10/23, 08:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/35621
-
Benni Mack commented on patch set 20 01/10/23, 09:01
Reply
DoneReply
DonePageRouter.php:245ReplyDone -
Benni Mack voted on patch set 20 V +1 CR +1 01/10/23, 09:02
-
Markus Klein voted on patch set 20 V +1 01/10/23, 09:14
-
Oliver Hader voted on patch set 20 V +1 01/10/23, 09:15
-
Benni Mack voted on patch set 20 V +2 CR +2 01/10/23, 09:15
-
Benni Mack merged patch set 21 01/10/23, 09:15