Review main Abandoned

69257[BUGFIX] Cache static TypoScript files

Andreas Kienast CR 0 V 0 Normal +64 −36 updated 08/08/21, 07:30

Commit message

To reduce I/O on uncached requests, available ext_typoscript_* files with
their respective contents are now stored in an cache.

Without cache, all installed extensions are iterated and checked for
several files at a specific location and read, which may cause a negative
I/O impact.

Review

  • Code review CR 0
  • Verified V 0

Issues 1

Files 2

Review log 1417

  1. Andreas Kienast uploaded patch set 1 05/25/21, 09:32

  2. core-ci voted on patch set 1 V -1 05/25/21, 09:35 Core CI is not happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/11821

  3. Andreas Kienast uploaded patch set 2 05/25/21, 09:39

  4. Christian Kuhn commented on patch set 1 05/25/21, 09:42

    You know I'm not a huge fan of adding caches ;) So, let's have some of the casual questions to see if we should really do this: * Is this properly evicted when adding/removing/loading/unloading extensions via extensionmanager and composer? That's kinda important to not introduce additional headaches, right? * Does it evict itself when content changes and/or is affected by 'clear all cache' button? * Why assets cache? * Could this impact the install tool which shouldn't rely on caches? * Which impact did you measure?
  5. core-ci voted on patch set 2 V +1 05/25/21, 09:49 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/11823

  6. Andreas Kienast commented on patch set 2 05/25/21, 09:54

    Reply

    1) Yes, I've updated and (re)installed an extension via EM, the uncached case took effect. 2) Content changes are not related at all to this functionality. The "Clear all cache" buttons, both in backend's top menu and Install Tool, clear the cache properly. 3) Because it's about static files on the disk, We store RequireJS stuff and similar there as well. 4) The Install Tool didn't trigger my breakpoint, so I assume we're safe here. 5) Unfortunately, I neither have a large website nor a server cast in wood to get any reasonable result. However, in theory this has a positive impact as only one file is used instead of checking all installed extension over and over again.
  7. Christian Kuhn commented on patch set 2 05/25/21, 10:05

    Reply

    Sounds good Andy. Thx. I still wonder if this has a positive impact since OS kernels will usually cache these accesses away itself by answering from disk buffers. It would thus be good to see if this has any real life impact.
  8. Simon Gilli commented on patch set 2 05/25/21, 10:15

    Unresolved

    For me this sounds more like a bugfix

    TemplateServiceTest.php:53 Unresolved

    Don't think this is intended
  9. Andreas Kienast uploaded patch set 3 05/25/21, 10:24

  10. Andreas Kienast commented on patch set 2 05/25/21, 10:26

    Reply

    I think semi-optimized (shared) hosting instances can take advantage from that additional cache, the reporter of the ticket also mentioned high I/O. Do you have any larger installation where you can test this patch?
  11. core-ci voted on patch set 3 V +1 05/25/21, 10:33 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/11826

  12. Simon Gilli commented on patch set 3 V +1 CR +1 05/25/21, 19:57

    Tested on one of my shared hosting servers @opsone, there is no real difference for the response time. But the patch works as expected.
  13. Christian Kuhn commented on patch set 3 V -1 05/26/21, 11:39

    Reply Unresolved

    Hey. I don't see the point in adding a cache if there is no measurable difference. Strictly, this change actually adds at least one I/O on empty caches. I doubt this change can be measured effectively since all OS kernels will eradicate any actual disk access - my point is that the kernel can do this type of optimizing much better than we can. Beside, on "semi-optimized (shared) hosting" (as andy wrote), there are many much more significant factors that can lead to high disk load: Most notably if the opcache does not exist or is not configured with enough ram, it will load the source files over and over from disk again. With hundreds if not thousands of source files, this kills performance much heavier than these I/O's. This will be worse if the kernel is heavily memory limited since the internal I/O buffers are too small then. Adding another file to this does not improve the situation in this case. All in all, I'm reluctant in adding another cache layer for non-cached pages accesses - the factor 100 or so in performance that is lost compared to a fully cached page has totally different reasons than these disk I/O's. Please change my mind by submitting according scenario data showing this change has significant positive impact.
  14. Benni Mack commented on patch set 3 06/14/21, 13:20

    @lolli I see a similar point like you, HOWEVER I could imagine we extract this exact logic into a separate class, which then uses the "core" cache. This makes the TemplateService class much smaller and both parts better to test. The decision if we want to use a cache or not could then be handled separately / looked at separately. I also *know for sure* that all other external files check for the timestamp ("rowSum" :D) to see if something has changed on *every* request, which is a bummer too, but could be separated at a later point to make all logic better to understand.
  15. Christian Kuhn commented on patch set 3 06/16/21, 20:36

    Reply Unresolved

    imho, the parser really finally needs a rewrite anyways - enabling this was one reason for my recent 'drop syntax highlighting from parser' patch ... so, as a totally selfish reason, i'd love to avoid adding additional layers meanwhile, we may then have to tackle/discuss when we improve performance with a much better cache layer that can't be done with current state of code later ;)
  16. Christian Kuhn commented on patch set 3 06/16/21, 20:43

    note: i think we can end up with a cached object tree per ts-file when we rewrite the parser, so no parsing of a file at all anymore, as soon as we properly rewrite the parser. that would also mean a file isn't parsed a second time, even if included at some other part of the page tree, for instance. this is currently not the case: the parser currently parses "everything" per page, no matter if sub-sections are included on other pages too and may have been parsed there already ... huge potential for more performance! this is why i'd love to avoid new layers at the moment - and i think a ts-parser rewrite *is* doable, we even have the helmich parser as example (i'd do some things differently, though, but it is a working implementation already).
  17. Andreas Kienast abandoned the change 08/08/21, 07:30