Review main Merged

69547[FEATURE] Fetch error pages via subrequests instead of HTTP

crell CR +2 V +2 Hard +265 −100 updated 09/11/21, 15:00

Commit message

There are many reasons to not issue a second HTTP request for error pages
and instead issue an in-process subrequest.  This patch makes that change,
and refactors PageContentErrorHandler a bit while there.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Benjamin Franzke +1
  • Verified V +2 Done
    • Benni Mack +2
    • Benjamin Franzke +1
    • core-ci +1 CI

Issues 1

Files 12

Review log 4979

Show 3154 earlier messages
  1. crell uploaded patch set 1 06/23/21, 21:54

  2. crell marked it work in progress 06/23/21, 21:55

  3. core-ci voted on patch set 1 V -1 06/23/21, 22:06 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12700

  4. crell uploaded patch set 2 06/23/21, 22:30

  5. core-ci voted on patch set 2 V +1 06/23/21, 22:41 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12701

  6. Christian Kuhn commented on patch set 2 06/24/21, 09:17

    That's a really great idea and I'm all in! Problem however is state. We're still dealing with state issues for sub requests in v11: We relatively recently finally killed (non constant) constant TYPO3_MODE (https://review.typo3.org/c/Packages/TYPO3.CMS/+/66948), this took us years. The constant is still there in v11 and still used in 3rd party extensions. static class state: See https://github.com/TYPO3/testing-framework/blob/master/Classes/Core/Functional/Framework/FrameworkState.php for more details. The testing framework uses sub requests for frontend functional tests in v11 (see https://github.com/TYPO3/testing-framework/blob/master/Classes/Core/Functional/FunctionalTestCase.php#L884). But I fear if we start doing that in non-testing core, people will run into trouble in real-life projects in v11 (which is a nasty thing in error handling in the first place): Too many rely on state and TYPO3_MODE in 3rd party extensions, so we probably shouldn't hard-wire a sub-request into the error handling right now. *if* we do, we'd probably at least should have a feature toggle, and we should be prepared to do some additional hacks similar to what FrameworkState.php of testing-framework does, most obvious one is a makeInstance singleton reset.
  7. Christian Kuhn commented on patch set 2 06/24/21, 13:35

    PageContentErrorHandler.php:141

    please feed your request to TYPO3\CMS\Frontend\Http\Application and not to the dispatcher - The frontend application is the entry point. See testing-framework sup-request handling as reference: https://github.com/TYPO3/testing-framework/blob/master/Classes/Core/Functional/FunctionalTestCase.php#L984
  8. crell uploaded patch set 3 06/24/21, 15:35

  9. core-ci voted on patch set 3 V -1 06/24/21, 15:38 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12711

  10. crell uploaded patch set 4 06/24/21, 15:43

  11. core-ci voted on patch set 4 V +1 06/24/21, 15:53 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12712

  12. crell uploaded patch set 5 06/24/21, 16:43

  13. core-ci voted on patch set 5 V -1 06/24/21, 16:47 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12716

  14. crell uploaded patch set 6 06/24/21, 16:49

  15. core-ci voted on patch set 6 V -1 06/24/21, 17:02 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12718

  16. core-ci voted on patch set 6 V +1 06/24/21, 17:51 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12718

  17. core-ci commented on patch set 6 06/24/21, 18:09 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12718

  18. crell uploaded patch set 7 06/25/21, 21:47

  19. core-ci voted on patch set 7 V -1 06/25/21, 21:59 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12753

  20. crell uploaded patch set 8 07/01/21, 22:41

  21. crell uploaded patch set 9 07/01/21, 22:43

  22. crell uploaded patch set 10 07/01/21, 22:46

  23. crell uploaded patch set 11 07/01/21, 22:56

  24. core-ci voted on patch set 8 V -1 07/02/21, 00:43 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12912

  25. core-ci voted on patch set 9 V -1 07/02/21, 00:48 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12913

  26. core-ci voted on patch set 10 V -1 07/02/21, 00:51 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12914

  27. core-ci voted on patch set 11 V -1 07/02/21, 00:57 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12915

  28. crell uploaded patch set 12 07/07/21, 20:55

  29. crell commented on patch set 12 07/07/21, 20:56

    This change is ready for review.
    So, there are some existing tests in place, but they're all quite buggy. The existing testing harness is incomplete. I'm unclear how or why but this version is now resulting in a deadlock. Maybe I can get more useful information faster from the CI server than local testing.

    PageContentErrorHandler.php:141 Reply

    Done
  30. crell uploaded patch set 13 07/07/21, 21:23

  31. core-ci voted on patch set 12 V -1 07/07/21, 22:57 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/13092

  32. core-ci voted on patch set 13 V -1 07/07/21, 23:25 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/13093

  33. crell uploaded patch set 14 07/09/21, 12:42

  34. core-ci voted on patch set 14 V -1 07/09/21, 14:45 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/13146

  35. Stephan Großberndt commented on patch set 14 07/13/21, 15:20

    SlugSiteRequestTest.php:684

    The name of this test suggests it is to test PHP Error handling activated but you call buildErrorHandlingConfiguration for 'Page' instead of 'PHP' now
  36. Benni Mack uploaded patch set 15 07/18/21, 16:01

  37. Benni Mack uploaded patch set 16 07/18/21, 16:01

  38. core-ci voted on patch set 15 V -1 07/18/21, 18:03 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/13449

  39. core-ci voted on patch set 16 V -1 07/18/21, 18:04 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/13450

  40. Benni Mack uploaded patch set 17 07/26/21, 14:32

  41. core-ci voted on patch set 17 V -1 07/26/21, 14:41 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/13762

  42. Benni Mack uploaded patch set 18 07/26/21, 16:45

  43. core-ci voted on patch set 18 V +1 07/26/21, 16:55 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/13780

  44. Benni Mack voted on patch set 18 V +1 CR +1 07/26/21, 17:00

  45. crell commented on patch set 18 CR +1 07/26/21, 17:49

    Thanks, Benni! I'm good with the changes here.
  46. Oli Bartsch commented on patch set 18 V -1 07/30/21, 23:08

    Hi, thanks for this patch. I have one problem so far: Having the feature flag on (which is the default), I get the html markup as string in the FE. Seems like the content-type header is somehow missing and only the markup is displayed. Another thing: When moving the 404 page out of the page tree on which I produce the 404 in the FE, I get following error: "Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/typo3/sysext/core/Classes/Routing/PageRouter.php on line 228" Not sure, but this reminds me of https://review.typo3.org/c/Packages/TYPO3.CMS/+/68438. Is there a reason why you do not use the cache for the new sub request functionality?

    PageContentErrorHandler.php:151

    Could be omitted as it's the default value

    PageContentErrorHandler.php:165

    Seems to be unused

    PageContentErrorHandler.php:195

    Prior check was a strict. So please make this strict, too.

    ShortcutAndMountPointRedirect.php:74

    Why is this moved into another middleware which is called earlier? Is this somehow related to this patch?
  47. crell commented on patch set 18 08/02/21, 14:10

    Reply

    Interesting. I get neither of these issues. The page renders for me, and I haven't run into a memory issue yet. If you have a memory size of 1 GB and it's still failing that sounds like an infinite loop to me, so I'll investigate further. I didn't cache the internal request mainly because it felt unnecessary. Caching the external request avoids an HTTP server invocation. The internal request just creates another cache clearing point to worry about.

    PageContentErrorHandler.php:151 Reply

    Done

    PageContentErrorHandler.php:165 Reply

    My gut feeling is this is somehow related to the infinite loop noted above and I need to use this method again. Will poke at it some more.

    PageContentErrorHandler.php:195 Reply

    Done

    ShortcutAndMountPointRedirect.php:74 Reply

    I think Benni did that one, so I defer to him.
  48. crell uploaded patch set 19 08/02/21, 14:51

  49. core-ci voted on patch set 19 V +1 08/02/21, 15:13 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14001

  50. crell uploaded patch set 20 08/02/21, 15:55

  51. core-ci voted on patch set 20 V +1 08/02/21, 16:28 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14009

  52. Benni Mack uploaded patch set 21 08/09/21, 18:04

  53. core-ci voted on patch set 21 V +1 08/09/21, 18:17 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14393

  54. Benni Mack uploaded patch set 22 08/13/21, 10:31

  1. core-ci voted on patch set 22 V +1 08/13/21, 10:45 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/14931

  2. crell uploaded patch set 23 08/20/21, 18:45

  3. core-ci voted on patch set 23 V -1 08/20/21, 19:01 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15346

  4. crell uploaded patch set 24 08/20/21, 20:37

  5. core-ci voted on patch set 24 V +1 08/20/21, 20:53 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15354

  6. Mathias Brodala commented on patch set 24 08/23/21, 06:57

    Just to be sure: https://github.com/pixelant/pxa_lpeh/issues/3 is not an issue here?
  7. Benni Mack commented on patch set 24 08/23/21, 19:26

    Reply

    ha, didn't know about this one. Yes, we need to "reset globals" :(
  8. crell uploaded patch set 25 08/25/21, 16:19

  9. core-ci voted on patch set 25 V +1 08/25/21, 16:36 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15614

  10. Benni Mack commented on patch set 25 V +1 CR +1 08/25/21, 21:42

    Thanks for the TSFE "parking". I'm still fine with this change, as it has a feature toggle so people with plugins with side-effects still have the possibility to opt-out (which we didn't have when we initially brought in this change in v10 for the first time).
  11. Benjamin Franzke commented on patch set 25 CR -1 08/26/21, 11:22

    MiddlewareStackResolver.php:49

    Why is this changed? PhpFrontendCache is required as `$this->cache->require()` is used in resolve().

    DefaultConfigurationDescription.yaml:202

    Probably added due to rebase? This has been removed with https://review.typo3.org/c/Packages/TYPO3.CMS/+/70544
  12. Benjamin Franzke uploaded patch set 26 08/26/21, 11:56

  13. Benjamin Franzke commented on patch set 26 08/26/21, 11:56

    PageContentErrorHandler.php:95

    Why don't we directly return the response? (return $response->withStatus($this->statusCode)) Having to copy the body seems like unneeded overhead(?) I know, this wrapping has been performed for the real-http-request-stuff as well, but I wonder if we could pass the response directly, or are there any reasons why we need to copy the body? If we'd really want a new response (e.g. if you want to drop headers on purpose), then we should create a new Response with `new Response(null, $this->statusCode)` and then attach the existing body buffer: $response->withBody($subResponse->getBody()); Actually, reading the previous code, it seems that *all* headers were transfered to the new request.

    PageContentErrorHandler.php:111

    I cannot manage to trigger this condition, did you?

    MiddlewareStackResolver.php:49 Reply

    Done

    DefaultConfigurationDescription.yaml:202 Reply

    Done
  14. core-ci voted on patch set 26 V +1 08/26/21, 12:15 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15668

  15. Simon Schaufelberger commented on patch set 26 V +1 CR +1 08/27/21, 17:11

    I created an extbase test extension, added some TypoScript settings and I can confirm that when I set the error handling to show content of a page with the plugin included, the TypoScript settings are there.
  16. Simon Schaufelberger commented on patch set 26 08/27/21, 17:13

    Reply

    I created an extbase test extension, added some TypoScript settings and I can confirm that when I set the error handling to show content of a page with the plugin included, the TypoScript settings are there.
  17. Benjamin Franzke uploaded patch set 27 08/27/21, 20:38

  18. Benjamin Franzke commented on patch set 27 08/27/21, 20:38

    PageContentErrorHandler.php:95 Reply

    Done. Regarding headers: nevermind, only content-type was passed before as well (don't know why). So I'll not change that part…

    PageContentErrorHandler.php:111 Reply

    Removed this now.

    PageContentErrorHandler.php:165 Reply

    It's unused and therefore dropped (with the hope to get the patch clean and easy to review). Feel free to rollback in case you need it again.

    FactoryConfiguration.php:28

    Since it defaults to true in DefaultConfiguration as well, it's not needed here. We'd only need this if we deciced that existing (upgraded) TYPO3 installations should not use subrequests (thus if we define subrequestPageErrors = false in DefaultConfiguration).

    ShortcutAndMountPointRedirect.php:74 Reply

    Basically because the workaround from ShortcutAndMountPointRedirect would now need to be added to the PageContentErrorHandler, and instead of doing that it is moved as a general cleanup to a middleware in response post-processing. That is fine imho – although it would of course be more useful if it would be moved into PrepareTypoScriptFrontendRendering (which itself acquires the lock) instead of TypoScriptFrontendInitialization. @benni: They both are executed one-after-the other. Any reason why you choosed to release in TypoScriptFrontendInitialization instead of PrepareTypoScriptFrontendRendering? Maybe because of middlewares that hook between those? Although that makes no sense, as they are both marked internal and marked to be avoided as referencepoint for ordering… Therefore I've now moved the call to releaseLocks into PrepareTypoScriptFrontendRendering. Please let me know if that's problematic.

    SlugSiteRequestTest.php:684 Reply

    good catch, fixed.
  19. core-ci voted on patch set 27 V -1 08/27/21, 20:59 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15752

  20. Benjamin Franzke uploaded patch set 28 08/28/21, 04:34

  21. core-ci voted on patch set 28 V +1 08/28/21, 04:52 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15760

  22. Benni Mack voted on patch set 28 V +2 CR +2 08/30/21, 06:51

  23. Benni Mack merged patch set 29 08/30/21, 06:51

  24. Torben Hansen commented on patch set 29 09/10/21, 09:04

    Reply

    While migrating an extension to TYPO3 11 I now experience exactly the same problem as described in the GitHub issue https://github.com/pixelant/pxa_lpeh/issues/3 (empty page when subrequest is used from Extbase). I created this core issue https://forge.typo3.org/issues/95174, since changes are already merged.
  25. Simon Schaufelberger commented on patch set 29 09/11/21, 15:00

    I've found a critical bug which I described here: https://forge.typo3.org/issues/95174. It would be awesome if somebody would come up with a solution as it seams like a DI issue