Review main Merged

60307[BUGFIX] Support concurrent requests without 503 responses

Benjamin Franzke CR +2 V +2 Normal +69 −130 updated 05/05/19, 21:02

Commit message

Instead of offloading the work to wait for the final page content,
concurrent requests now wait for the real page content to be
rendered (and deliver the content from cache once ready) instead
of sending a 503 response code and the famous "Page is being
generated" message.

The logic to wait for the rendered page (lock and wait) is already
there thanks to the page locking, but was deliberately circumvented
by the temporary page cache content before to get rid of waiting
requests in high-load situations.

This approach to simply skip the temporary cache and wait for the
renderer to finish has been tested using the skip_page_is_being_generated
extension in wild for 3 years (for TYPO3 v6, v7, v8 and v9).

Note: In case the increased number of waiting requests has a negative
impact on sites with high server load, a additional proxy cache should be
considered in front of the server to make sure clients are served a valid
response without waiting until new content is ready.

The motivation for this change:
The 503 status code together with the "Page is being generated message"
does not only occur for slow or high traffic sites. It will be displayed
even for two concurrent requests, no matter how fast the page rendered
or how low the current traffic is.
The requests only need to (nearly) arrive at the same time. This can
easily be reproduced using two parallel curl requests:
for i in {1..2}; do curl -sv https://doma.in/foo |& grep '^< HTTP'& done
There would be one "503 Service unavailable" response when /foo has not
yet been rendered to the cache before.

An explanation for the releaseLock('pagesection') addition in
TSFE::getFromCache(): This has been added as the pagesection lock – which
is acquired in TemplateService::start() – was implicitly released in
setPageCacheContent() before. Now this would block concurrent rendering
for pages with $_GET-aware plugins, therefore we release the pagesection
lock early, after the pagesection cache has been generated.

Review

  • Code review CR +2 Done
    • Benjamin Franzke +2
    • Georg Ringer +1
    • Markus Klein +1
    • Oliver Hader +1
    • Helmut Hummel +1
    • Benni Mack +1
  • Verified V +2 Done
    • Benjamin Franzke +2
    • Oliver Hader +1
    • Helmut Hummel +1
    • TYPO3com +1 CI

Issues 1

Files 4

Review log 3440

Show 1315 earlier messages
  1. Benjamin Franzke uploaded patch set 1 03/22/19, 09:03

  2. TYPO3com voted on patch set 1 V +1 03/22/19, 09:17 Completed build in 13m 39s on Fri, 22 Mar, 10:17 AM Test Summary: 47201 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-32882

  3. Benjamin Franzke updated the commit message in patch set 2 03/22/19, 09:30

  4. TYPO3com voted on patch set 2 V +1 03/22/19, 09:44 Completed build in 13m 41s on Fri, 22 Mar, 10:44 AM Test Summary: 47201 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-32883

  5. Markus Klein commented on patch set 2 CR -1 03/22/19, 21:59

    I have to disagree. The temporary page content is specifically there for offloading requests. If you keep them hanging in the locks, you end up with filling up your used TCP sockets on the server until you reach the kernel limit. When there is really high load and the whole FE cache was flushed and you need to basically regenerate all of them at the same time and each of them maybe needs a few seconds, you have basically all requests for all pages "hanging" on your server. Therefore I suggest keeping this logic as it is the only known way to me to tell the browser "please try again a little later, I'm working hard to get your content ready"
  6. Markus Klein commented on patch set 2 03/22/19, 22:03

    Unresolved

    There seems something wrong then. Because if the page is in cache this content never appears. Yes, it would probably be okay to have a second request waiting instead of off-loading them, but for a sane solution we need proper "pending request" counting.
  7. Benjamin Franzke marked it work in progress 03/25/19, 15:50

  8. Jonas Eberle commented on patch set 2 03/27/19, 20:18

    I support this patch because I think it is the right way to do it. The current way tells the user that he shall reload the page in some seconds. But a 503 is mostly not understood as what we want to communicate here ("come back in 10 seconds"): It means that there is maintenance or an upstream cannot be reached, so a real technical problem. 503 is understood as "check back in 1 hour or a day" (we actually send the Retry-After: 3600 header right now but that's a) a whole hour and b) mostly not implemented in crawlers). So most crawlers will just mark that crawler run as failed. Currently page previews for slack/facebook/twitter/... are simply not generated in that circumstance because the crawlers will not retry after some seconds but fail completely. Indexers like Google will ignore it and keep the old content. Google search console will probably show that as "page errors (500)". Also mind that this might also happen for cached AJAX content. Actually in the current scheme we would have to adapt our frontend JavaScript to cater for that. Rather let it wait until its generated (some seconds). The perfect way would be to return a 503 if the request could not be fulfilled within 10s or so. But I think that is technically not feasible so I rather have this new approach of waiting and returnung sane content with 200. If a page would take really long to generate I think it is OK to just hit the timeout. This eats up connections/worker threads, but so be it.
  9. Jonas Eberle commented on patch set 2 V +1 CR +1 03/27/19, 20:41

    works like a charm
  10. Markus Klein commented on patch set 2 03/28/19, 07:32

    A possible improvement would be to keep only a certain number of clients waiting and offload only those, which do not fit anymore. We would have to: - implement the locking in a way that those waiting clients actually hang in a blocking lock then - some way to determine the right number of clients which could be held waiting Problem with the last point is that this is almost impossible to find out. Consider a shared host with 1000 TYPO3 pages with varying load. The sever may be able to handle 1000 TCP connections at the same time, before dropping new connections entirely. How to distribute those connections in a fair manner? Who can one instance determine its "share" of TCP connections, which may be kept "on hold" while prefilling caches? Any idea is welcome!
  11. Jonas Eberle commented on patch set 2 03/28/19, 09:05

    I'd rather have that handled by a frontend proxy if really needed. I don't think it is our business. This new behaviour would make it easier for caching proxies. This change gives predictable results (think of monitoring your site, too, where you expect 200) and strips (IMHO unnecessary) complexity from the CMS thus making it easier to maintain. It also makes error codes less likely. Error codes make SEO people and clients nervous. This specific one is hard to understand without TYPO3 knowledge if an admin spots it in his logfiles. On top it improves chances to rely on well understood standards (503 is a difficult animal). I would see this as a step forward.
  12. Markus Klein commented on patch set 2 03/29/19, 07:38

    > Patch Set 2: > > I'd rather have that handled by a frontend proxy if really needed. I don't think it is our business. This new behaviour would make it easier for caching proxies. > > This change gives predictable results (think of monitoring your site, too, where you expect 200) and strips (IMHO unnecessary) complexity from the CMS thus making it easier to maintain. > > It also makes error codes less likely. Error codes make SEO people and clients nervous. This specific one is hard to understand without TYPO3 knowledge if an admin spots it in his logfiles. > > On top it improves chances to rely on well understood standards (503 is a difficult animal). > > I would see this as a step forward. First of all, "(503 is a difficult animal)" is something that I simply do not accept. HTTP is a standard and is there for ages. Everyone having troubles with learning what it all is about => book a training. If shitty SEO Software can't deal with it, drop it. Google can deal with it. I'm all in for any simplification, don't get me wrong. But I suggest that we first look into the history, why this all was really introduced. Maybe even talk to those people who introduced it, if somehow possible. Once we a clear picture and know all the arguments, we can lay out a plan to reduce complexity. Regarding the idea to push "the load problem" to a proxy. Well, many (if not most) TYPO3 sites do not use any service before the actual web server. So in the worst case all the connections pile up until the TCP queue is full. All subsequent requests will then experience the even more critical situation of "host not reachable" or "connection refused". UX wise I consider the current message "we are working hard to get your page ready for you" much better than a blunt "connection refused". I still tend to optimize pages for people and not for machines (alias bots ;-))
  13. Jonas Eberle commented on patch set 2 03/29/19, 11:24

    This is a) my opinion and it is b) backed by knowledge (HTTP) and experience. The current system is broken. This change gets us back to sanity. Currently on parallel requests that take 1s to render: 503 (T+0) / 200 (T+1s) / 200 (T+1s) / 200 (T+1s) / ... -> so even IF the 503-approach was a good idea, it is badly implemented. And 503 is a bad idea for "come back in some seconds", even IF the Retry-After-header WAS set to a shorter value. All of that is not true so what's the point? new: 200 (T+1s) / 200 (T+1s) / 200 (T+1s) / 200 (T+1s) / ... -> this is easy to handle by anything downstream Please don't take it personal. I don't expect a reply. Let other people decide if that should go in.
  14. Markus Klein commented on patch set 2 CR 0 03/29/19, 12:05

    > Patch Set 2: > > This is a) my opinion and it is b) backed by knowledge (HTTP) and experience. > The current system is broken. This change gets us back to sanity. > > Currently on parallel requests that take 1s to render: > 503 (T+0) / 200 (T+1s) / 200 (T+1s) / 200 (T+1s) / ... > -> so even IF the 503-approach was a good idea, it is badly implemented. And 503 is a bad idea for "come back in some seconds", even IF the Retry-After-header WAS set to a shorter value. All of that is not true so what's the point? > > new: > 200 (T+1s) / 200 (T+1s) / 200 (T+1s) / 200 (T+1s) / ... > -> this is easy to handle by anything downstream > > Please don't take it personal. I don't expect a reply. Let other people decide if that should go in. of course it can be a bad implementation. but I'm currently not in the position (due to lack of knowledge of the exact original intentions) to decide that. I'm not blocking this patch (actually I cannot even do that), I'm just sharing my concerns in this regard. Keep in mind that I also can speak of some experience, when it comes to lowlevel dev stuff in core. It happened not only once that "the idea was so perfect" until the first regression comes in after a release, where all of a sudden things become clear, why the idea was maybe far from perfect. I'll refrain from further disturbing you in this regard of course.
  15. Markus Klein commented on patch set 2 03/29/19, 12:31

    one point more of knowledge: this very functionality dates back to Kasper's initial commit! :-) 16 years.
  1. Markus Klein commented on patch set 2 03/29/19, 12:36

    one more note on the discussion with Jonas: You assume 1s creation time here, which I consider way to optimistic. Our largest pages sometimes have 5s and more, depending on the complexity of the pages. So again, in 5s there can be quite some requests that pile up.
  2. Helmut Hummel commented on patch set 2 V +1 CR +1 04/30/19, 17:46

    Why is it still WIP? Because of RST missing?
  3. Markus Klein commented on patch set 2 04/30/19, 18:44

    I had another longer conversation with Helmut and also checked with Benni. I am fine with removing the temp cache stuff, but we can go even further then and remove those locking parts that are not needed anymore. Therefore I would extend this patch. Moreover we should include the little change of Helmut in acquireLock of https://review.typo3.org/c/Packages/TYPO3.CMS/+/60634/1/typo3/sysext/frontend/Classes/Controller/TypoScriptFrontendController.php
  4. Markus Klein uploaded patch set 3 04/30/19, 21:43

  5. Markus Klein commented on patch set 3 04/30/19, 21:51

    I took the liberty to add a lot of documentation to the acquireLock function and to incorporate a suggested change from Helmut regarding the releaseLock of pagesection cache. Moreover please see my thoughts on the retry-rate of the page lock

    TypoScriptFrontendController.php:3695

    I lowered that one a lot tooo, but if I follow the math below, we can probably safely go back to 100ms again. Assuming 5 requ/s (on the samge URL!) is on average every 200ms a new request. So if the page generation runs 200 to 1000ms (or even longer) it should be really sufficient to check every 100ms.
  6. TYPO3com voted on patch set 3 V +1 04/30/19, 21:53 Completed build in 10m 31s on Tue, 30 Apr, 11:53 PM Test Summary: 47475 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-33680

  7. Helmut Hummel commented on patch set 3 CR -1 05/01/19, 10:47

    The change from https://review.typo3.org/#/c/Packages/TYPO3.CMS/+/60634/ should imho be removed from here

    TypoScriptFrontendController.php:1838

    Can we please treat this change (proposed here: https://review.typo3.org/#/c/Packages/TYPO3.CMS/+/60634/) as individual change and not mix both together? The fix should go into other branches as well, while the cleanup here should only go to master
  8. Helmut Hummel commented on patch set 3 05/01/19, 11:23

    As a side note: the change here: https://review.typo3.org/#/c/Packages/TYPO3.CMS/+/58582/ caused the temporary cache being deleted by the first hit that completes the request, thus since then the temp cache never served the purpose of requests not waiting for locks but delivering the temp content until page rendering completes
  9. Markus Klein commented on patch set 3 05/03/19, 06:55

    TypoScriptFrontendController.php:1838 Reply

    Fine with me, but I would also merge this change all the way down to 8.7
  10. Markus Klein commented on patch set 4 05/03/19, 07:06

    Published edit on patch set 3.
  11. TYPO3com voted on patch set 4 V +1 05/03/19, 07:21 Completed build in 14m 51s on Fri, 3 May, 09:21 AM Test Summary: 47475 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-33701

  12. Helmut Hummel commented on patch set 4 05/05/19, 11:02

    Sorry, I was wrong. My change here: https://review.typo3.org/#/c/Packages/TYPO3.CMS/+/60634/ does not really fix the issue, so Markus was right to include some changes from there here and target this change to all branches. I will restore therefore restore patch set 3 as suggested by Markus
  13. Helmut Hummel uploaded patch set 5 05/05/19, 11:03

  14. TYPO3com voted on patch set 5 V +1 05/05/19, 11:17 Completed build in 14m 08s on Sun, 5 May, 01:17 PM Test Summary: 47475 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-33715

  15. Benjamin Franzke uploaded patch set 6 05/05/19, 14:48

  16. TYPO3com voted on patch set 6 V +1 05/05/19, 15:01 Completed build in 12m 42s on Sun, 5 May, 05:01 PM Test Summary: 47475 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-33717

  17. Benjamin Franzke commented on patch set 6 V +1 CR +1 05/05/19, 15:01

    Thanks for adding the improvements from Helmuts separate patch into this one. I initially thought they would be better suited as a separate optimization commit, but they are fine to be contained in this patch as the releaseLock() removal in setPageCacheContent() really belongs to this change (it's purpose was to ensure the temporary message is shown instead of blocking the concurrent processes, which is what we want now). The addition in getFromCache() is to not regress in term of concurrent $_GET-variable aware plugin rendering. I've added explanations for these changes to the commit message. Additionally I added an Important.rst now (which was the reason for this patch to be marked [WIP]). From my point of view we're ready to merge. Please test and revote (you know, tomorrow may already be merge freeze…)

    TypoScriptFrontendController.php:3695 Reply Unresolved

    Yup, I think 100ms is fine for now, especially as we want to backport we should rather revisit this timeout later, only for master. (Although I'd really like to see no usleep()'s at all, but that's another story).
  18. Georg Ringer voted on patch set 6 CR +1 05/05/19, 17:47

  19. Oliver Hader voted on patch set 6 V +1 CR +1 05/05/19, 18:26

  20. Helmut Hummel voted on patch set 6 V +1 CR +1 05/05/19, 18:59

  21. Markus Klein voted on patch set 6 CR +1 05/05/19, 19:26

  22. Benni Mack commented on patch set 6 CR +1 05/05/19, 20:24

    go for it.
  23. Benjamin Franzke voted on patch set 6 V +2 CR +2 05/05/19, 20:59

  24. Benjamin Franzke merged patch set 7 05/05/19, 21:00

  25. Benjamin Franzke commented on patch set 7 05/05/19, 21:02

    Manual cherry-pick for v8 and v9 required. Please test the backports: v9: https://review.typo3.org/c/Packages/TYPO3.CMS/+/60656 v8: https://review.typo3.org/c/Packages/TYPO3.CMS/+/60660 Thank you!