Review main Merged

61645[FEATURE] Add cache:warmup console command

Benjamin Franzke CR +2 V +2 Hard +696 −47 updated 08/30/21, 12:33

Commit message

A cache:warmup CLI command is added that is safe to be
executed during deployment or development mode.

It is made extra sure that the command will be executable,
even if all caches are stale or even broken. That means this
command is able to repair broken di, localconf and tca caches.
Technically this became due to preparations for lowlevel
cli commands in #86248.

Caches that will be warmed up by TYPO3 core:
 + DI
 + ext_localconf
 + TCA
 + ext_tables
 + middleware stack
 + PackageManager
 + sites-configuration
 + expressionLanguageProviders
 + BackendRoutes
 + l10n (parses all xlf files, and creates cache for all languages)
 + assets: BackendIcons
 + dashboard: DashboardPresets
 + dashboard: WidgetGroups

Out of scope or impossible to be warmed:
 * Frontend warmup:
   Can be providedby extensions that listen for the
   CacheWarmupEvent
 * RequireJS (assets cache): Often applies with context-conditions
   therefore not reliably warmable.
 * Extbase – better of for a separate patch
   intesting cache entries would be: RequestHandlers, ClassSchemata
   and PersistenceClasses(?)
 * Warmup of Fluid templates:
   It is impossible to auto-detect whether all files in
   Resources/{Templates,Layouts,Partials} are Fluid files.
   Could by added later on with an explicit registry that
   allows to define which templates are suitable for warmup.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Helmut Hummel +1
  • Verified V +2 Done
    • Benni Mack +2
    • Helmut Hummel +1
    • core-ci +1 CI

Issues 1

Files 21

Show 1 more file

Review log 4886

Show 3261 earlier messages
  1. Benjamin Franzke uploaded patch set 1 09/08/19, 22:09

  2. TYPO3com voted on patch set 1 V -1 09/08/19, 22:10 Completed build in 13s on Sun, 8 Sep, 10:10 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36198

  3. Benjamin Franzke uploaded patch set 2 09/08/19, 22:24

  4. TYPO3com voted on patch set 2 V -1 09/08/19, 22:24 Completed build in 11s on Sun, 8 Sep, 10:24 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36199

  5. Benjamin Franzke uploaded patch set 3 09/08/19, 22:37

  6. TYPO3com voted on patch set 3 V -1 09/08/19, 22:37 Completed build in 10s on Sun, 8 Sep, 10:37 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36200

  7. Benjamin Franzke uploaded patch set 4 09/08/19, 22:44

  8. TYPO3com voted on patch set 4 V -1 09/08/19, 22:45 Completed build in 12s on Sun, 8 Sep, 10:45 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36201

  9. Benjamin Franzke wrote 09/19/19, 12:05

    Topic set to di
  10. Benjamin Franzke uploaded patch set 5 09/23/19, 14:52

  11. Benjamin Franzke updated the commit message in patch set 6 09/23/19, 14:54

  12. TYPO3com voted on patch set 5 V +1 09/23/19, 15:04 Completed build in 11m 46s on Mon, 23 Sep, 03:04 PM Test Summary: 47737 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36694

  13. TYPO3com voted on patch set 6 V +1 09/23/19, 15:10 Completed build in 12m 19s on Mon, 23 Sep, 03:10 PM Test Summary: 47737 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-36695

  14. Benjamin Franzke uploaded patch set 7 02/04/21, 16:59

  15. TYPO3com voted on patch set 7 V -1 02/04/21, 17:00 Completed build in 18s on Thu, 4 Feb, 05:00 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9267

  16. core-ci voted on patch set 7 V +1 02/04/21, 17:10 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/383

  17. Benjamin Franzke wrote 02/04/21, 18:54

    Topic di removed
  18. Benjamin Franzke wrote 02/04/21, 18:54

    Topic set to cli
  19. Benjamin Franzke uploaded patch set 8 02/05/21, 00:34

  20. TYPO3com voted on patch set 8 V -1 02/05/21, 00:34 Completed build in 18s on Fri, 5 Feb, 12:34 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9272

  21. core-ci voted on patch set 8 V -1 02/05/21, 00:43 Core CI is not happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/389

  22. Benjamin Franzke uploaded patch set 9 02/05/21, 09:15

  23. TYPO3com voted on patch set 9 V -1 02/05/21, 09:15 Completed build in 19s on Fri, 5 Feb, 09:15 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9285

  24. core-ci voted on patch set 9 V +1 02/05/21, 09:24 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/403

  25. Benjamin Franzke uploaded patch set 10 02/05/21, 10:05

  26. TYPO3com voted on patch set 10 V -1 02/05/21, 10:05 Completed build in 18s on Fri, 5 Feb, 10:05 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9288

  27. core-ci voted on patch set 10 V +1 02/05/21, 10:17 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/406

  28. Benjamin Franzke uploaded patch set 11 02/05/21, 13:51

  29. TYPO3com voted on patch set 11 V -1 02/05/21, 13:52 Completed build in 17s on Fri, 5 Feb, 01:52 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9297

  30. core-ci voted on patch set 11 V +1 02/05/21, 14:10 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/414

  31. Benjamin Franzke uploaded patch set 12 02/09/21, 16:31

  32. TYPO3com voted on patch set 12 V -1 02/09/21, 16:32 Completed build in 21s on Tue, 9 Feb, 04:32 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9405

  33. core-ci voted on patch set 12 V +1 02/09/21, 16:50 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/542

  34. Benjamin Franzke uploaded patch set 13 02/09/21, 18:28

  35. TYPO3com voted on patch set 13 V -1 02/09/21, 18:28 Completed build in 20s on Tue, 9 Feb, 06:28 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9406

  36. core-ci voted on patch set 13 V -1 02/09/21, 18:29 Core CI is not happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/544

  37. Benjamin Franzke uploaded patch set 14 02/09/21, 22:37

  38. TYPO3com voted on patch set 14 V -1 02/09/21, 22:38 Completed build in 17s on Tue, 9 Feb, 10:37 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9421

  39. core-ci voted on patch set 14 V +1 02/09/21, 22:48 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/559

  40. Benjamin Franzke uploaded patch set 15 02/10/21, 05:30

  41. TYPO3com voted on patch set 15 V -1 02/10/21, 05:31 Completed build in 20s on Wed, 10 Feb, 05:31 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9430

  42. core-ci voted on patch set 15 V +1 02/10/21, 05:55 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/570

  43. Benjamin Franzke rebased onto patch set 16 02/19/21, 20:29

  44. TYPO3com voted on patch set 16 V -1 02/19/21, 20:30 Completed build in 21s on Fri, 19 Feb, 08:29 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-9644

  45. core-ci voted on patch set 16 V +1 02/19/21, 20:47 Core CI is happy: https://core.b13.com/typo3-core/typo3.cms/-/pipelines/864

  46. Benjamin Franzke uploaded patch set 17 04/29/21, 15:15

  47. core-ci voted on patch set 17 V +1 04/29/21, 15:27 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/10912

  48. Benjamin Franzke wrote 04/30/21, 15:21

    Hashtag added: cache
  49. Andreas Kienast commented on patch set 17 CR -1 04/30/21, 16:00

    CacheWarmupEvent.php:33

    Return value must be of type 'string', 'array' returned

    CacheWarmupCommand.php:59

    : void

    IconRegistry.php:810

    Why do we need to backup the possibly stale state?

    CacheWarmer.php:56

    Is this todo still valid?
  50. Benjamin Franzke uploaded patch set 18 04/30/21, 16:19

  51. Benjamin Franzke commented on patch set 18 04/30/21, 16:19

    CacheWarmupEvent.php:33 Reply

    Done

    CacheWarmupCommand.php:59 Reply

    Done

    IconRegistry.php:810 Reply

    I basically wanted to be as side-effect free as possible. There is no possible bug though (right now) I can think of, if we wouldn't do this.

    CacheWarmer.php:56 Reply

    Unfortunately yes, don't have API to force renewal for such files yet. I thought it's good to have warmup at least (helps for the deployment scenario), but maybe I should drop l10n warmup for now anyway? I'm fine either way…
  52. core-ci voted on patch set 18 V +1 04/30/21, 16:30 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/10975

  53. Christian Kuhn commented on patch set 18 CR -1 04/30/21, 18:17

    Maybe add a basic functional test around the main command to verify the system does not fatal or similar? I'm asking since this commands hooks/registers into quite some modules, so chances are something breaks if they are refactored. Having a basic functional would ensure your work can't break easily, and cli stuff isn't something dev's test often during development.
  54. Benni Mack commented on patch set 18 05/01/21, 13:20

    Hey bnf, thanks for this change. I don't know really how I would be able to use this when building packages or deploying this (yet) via CI or natively yet. I mean, it's great that I don't have to warm up each of them "during runtime" but the preferred solution would be to use this after TYPO3 Console's "cache:flush" command, right? e.g. packaging via CI and then deploying an artefact, but symlinking folders such as var/ - then I would do cache:warmup after deployment? 1. CI: Package everything 2. Deploy via deployer (for example) 3. On target env: change symlinks 4. On target env: cache:flush 5. On target env: cache:warmup?
  55. Christian Kuhn commented on patch set 18 05/01/21, 16:56

    Reply

    maybe even warmup on CI and deploy the warmed-up core-cache artifact? possible? or are there things like absolute paths involved?
  56. Benni Mack commented on patch set 18 05/01/21, 17:26

    Reply

    how? because it's all in var/cache/? maybe this would need to be made deployable as well...
  57. Benjamin Franzke commented on patch set 18 05/02/21, 12:11

    Reply

    Idea would be to run `vendor/bin/typo3 cache:warmup system` on the livesystem before doing a symlink switch to a new release, but already *on* the new release. That means the new release would go live with caches warmed up. A full *cache:flush* (run after symlink switch via TYPO3 console) would of course "ruin" that benefit. Therefore I would only flush page caches. To summarize: It is to be used during deployment, on the live system, in the new release folder, before switching the new release live: * composer install * bin/typo3 cache:warmup system * change symlink * bin/typo3 cache:clear pages (this one is TODO, but I'm working on that as well) Te idea is to warmup all file-related caches *before* the symlink switch to a new release and to *only* flush database/frontend (shared) caches after the symlink. Why file-related caches before the switch? These caches should not be shared between releases. In other words, this implies that var/ is *not* shared between releases. Only certian subdirectories: * var/session * var/log * var/lock * (and possibly) var/charset I don't want to recommend `cache:flush` + `cache:warmup` as that means there's a short gap where new requests on the new release will try to warmup caches, which this is about to prevent. Final note: Creation on CI is not an option as TYPO3 hashes absolute paths into cache hashes, so it really needs to be executed in the context of the final release folder.
  58. Benni Mack commented on patch set 18 05/02/21, 13:18

    Reply

    Thanks. I guess we should document this properly then in the Feature.rst to let people know how to benefit from it in their usual setups?
  59. Benni Mack uploaded patch set 19 06/14/21, 10:24

  60. core-ci voted on patch set 19 V +1 06/14/21, 10:34 Core CI is happy: https://gitlab.typo3.org/TYPO3/CI/cms/-/pipelines/12396

  61. Benjamin Franzke uploaded patch set 20 08/25/21, 12:34

  1. Benjamin Franzke commented on patch set 20 08/25/21, 12:34

    Reply

    Done
  2. Benjamin Franzke uploaded patch set 21 08/25/21, 12:35

  3. core-ci voted on patch set 20 V +1 08/25/21, 12:56 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15591

  4. core-ci voted on patch set 21 V -1 08/25/21, 13:08 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15592

  5. Benjamin Franzke uploaded patch set 22 08/25/21, 13:42

  6. Benjamin Franzke commented on patch set 22 08/25/21, 13:42

    Reply

    Done
  7. core-ci voted on patch set 22 V +1 08/25/21, 13:55 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15603

  8. Benjamin Franzke commented on patch set 22 08/26/21, 09:35

    IconRegistry.php:810 Reply

    Marking this resolved, I think that bit of extra safety shouldn't be an issue.

    CacheWarmer.php:56 Reply

    FYI: I would like to keep it as todo, therefore I'll keep this as is for now.
  9. Benjamin Franzke uploaded patch set 23 08/27/21, 11:37

  10. core-ci voted on patch set 23 V -1 08/27/21, 11:48 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15718

  11. Benjamin Franzke uploaded patch set 24 08/27/21, 12:42

  12. core-ci voted on patch set 24 V -1 08/27/21, 13:03 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15722

  13. Helmut Hummel rebased onto patch set 25 08/27/21, 15:27

  14. Benjamin Franzke uploaded patch set 26 08/27/21, 15:29

  15. core-ci voted on patch set 26 V +1 08/27/21, 15:48 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15736

  16. core-ci voted on patch set 25 V -1 08/27/21, 15:50 Core CI is not happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15735

  17. Benjamin Franzke uploaded patch set 27 08/27/21, 17:19

  18. core-ci voted on patch set 27 V +1 08/27/21, 17:32 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15743

  19. Helmut Hummel commented on patch set 27 V +1 CR +1 08/28/21, 20:23

    This is an amazing achievement and improvement! Thanks a bunch! Works like a charm

    CacheWarmupCommandTest.php:105 Unresolved

  20. Helmut Hummel rebased onto patch set 28 08/28/21, 20:24

  21. Helmut Hummel voted on patch set 28 V +1 CR +1 08/28/21, 20:26

  22. core-ci voted on patch set 28 V +1 08/28/21, 20:42 Core CI is happy: https://gitlab.typo3.org/typo3/CI/cms/-/pipelines/15786

  23. Benni Mack voted on patch set 28 V +2 CR +2 08/30/21, 06:12

  24. Benni Mack merged patch set 29 08/30/21, 06:12

  25. Helmut Hummel commented on patch set 29 08/30/21, 12:33

    \o/