Review main Merged

78787[TASK] Concurrently fetch URLs in HashProxy

Christian Eßl CR +2 V +2 Normal +93 −24 updated 04/24/23, 08:22

Commit message

Avoid too much I/O blocking when compiling URLs in
`HashProxy` by doing concurrent asynchronous fetch
requests with the GuzzleHttp client.

In addition, results of previously resolved URLs
are now stored in `assets` cache for one week.

Review

  • Code review CR +2 Done
    • Andreas Kienast +2
    • Georg Ringer +1
  • Verified V +2 Done
    • Andreas Kienast +2
    • Georg Ringer +1
    • core-ci +1 CI

Issues 1

Files 9

Review log 3147

Show 1522 earlier messages
  1. Christian Eßl uploaded patch set 1 04/21/23, 14:58

  2. Christian Eßl uploaded patch set 2 04/21/23, 15:09

  3. core-ci voted on patch set 1 V -1 04/21/23, 15:16 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41044

  4. Christian Eßl rebased onto patch set 3 04/21/23, 15:18

  5. Christian Eßl uploaded patch set 4 04/21/23, 15:20

  6. core-ci voted on patch set 2 V -1 04/21/23, 15:26 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41045

  7. core-ci voted on patch set 3 V -1 04/21/23, 15:26 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41046

  8. Oliver Klee commented on patch set 4 04/21/23, 15:48

    HashProxy.php:178

    Please let's have more details about the parameter and the return value as PHPDoc annotations both for the benefit of the human reader as well as PHPStan. The parameter looks like `list<string>`, and for the return value, I'm not quite sure.
  9. core-ci voted on patch set 4 V -1 04/21/23, 15:51 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41048

  10. Christian Eßl uploaded patch set 5 V -1 04/21/23, 16:06

  11. Christian Eßl uploaded patch set 6 04/21/23, 16:07

  12. Christian Eßl commented on patch set 6 04/21/23, 16:08

    HashProxy.php:178 Reply

    Done
  13. Christian Eßl uploaded patch set 7 04/21/23, 16:14

  14. core-ci voted on patch set 6 V +1 04/21/23, 16:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41054

  15. Oliver Klee commented on patch set 7 04/21/23, 16:29

    Do you have any numbers on how much this improves things?
  16. core-ci voted on patch set 7 V +1 04/21/23, 16:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41056

  17. Christian Eßl commented on patch set 7 04/21/23, 16:55

    Reply

    At the moment it is just theoretical, as the resource contents of files etc. afaik aren't compiled yet in the Content-Security-Policy feature. But when HashProxies are fully used in the future, it would depend on the number of files in the frontend/backend, that use such a hash source.
  18. Oliver Hader commented on patch set 7 04/21/23, 20:36

    HashProxy.php:179

    What do you think about using `GuzzleHttp\Pool` instead to limit the max. number of concurrent request (to e.g. 10 or 20)?
  19. Oliver Hader commented on patch set 7 04/21/23, 20:57

    I don't know, how intense this will be used in TYPO3 projects... but I think introducing a local cache for "hash per URI" would make sense to avoid those recurring URI fetches.
  20. Oliver Hader uploaded patch set 8 V +1 04/21/23, 21:00

  21. core-ci voted on patch set 8 V +1 04/21/23, 21:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41065

  22. Oliver Hader commented on patch set 8 04/21/23, 21:12

    How to test: see https://forge.typo3.org/issues/100708#note-9 Testes with `hey`: * before: Average: 0.9083 secs, Requests/sec: 1.9180 * after: Average: 0.7375 secs, Requests/sec: 2.2789
  1. Oliver Klee wrote 04/21/23, 21:54

    Removed cc Oliver Klee.
  2. Oliver Hader commented on patch set 8 04/21/23, 22:33

    I've been working on a cache (much better results): `Average: 0.0365 secs, Requests/sec: 54.8154`
  3. Oliver Hader uploaded patch set 9 V +1 04/21/23, 22:34

  4. Oliver Hader uploaded patch set 10 04/21/23, 22:35

  5. Oliver Hader commented on patch set 10 V +1 CR +1 04/21/23, 22:37

    Reply

    Done

    HashProxy.php:179 Reply

    skipped
  6. core-ci voted on patch set 10 V -1 04/21/23, 22:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41068

  7. core-ci voted on patch set 9 V -1 04/21/23, 22:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41067

  8. core-ci commented on patch set 10 04/21/23, 23:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41068

  9. Oliver Hader uploaded patch set 11 CR +1 V +1 04/21/23, 23:07

  10. core-ci voted on patch set 11 V -1 04/22/23, 00:15 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41071

  11. Oliver Hader rebased onto patch set 12 V -1 04/22/23, 05:50

  12. core-ci voted on patch set 12 V -1 04/22/23, 06:06 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41078

  13. Oliver Hader uploaded patch set 13 V -1 04/22/23, 06:27

  14. core-ci voted on patch set 13 V +1 04/22/23, 06:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41080

  15. Oliver Hader rebased onto patch set 14 V +1 04/22/23, 08:14

  16. Oliver Hader commented on patch set 15 04/22/23, 08:20

    Published edit on patch set 14.
  17. core-ci voted on patch set 14 V +1 04/22/23, 08:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41082

  18. core-ci voted on patch set 15 V +1 04/22/23, 08:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41083

  19. Oliver Hader uploaded patch set 16 V +1 04/24/23, 07:24

  20. Georg Ringer voted on patch set 16 V +1 CR +1 04/24/23, 07:34

  21. core-ci voted on patch set 16 V +1 04/24/23, 08:02 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41121

  22. Andreas Kienast commented on patch set 16 04/24/23, 08:05

    HashProxy.php:184

    If we'd filter out null values here, we could drop the whole null handling in the loop and in the `compileHashValues()` method (line 177).
  23. Oliver Hader commented on patch set 16 04/24/23, 08:09

    HashProxy.php:184 Reply

    If we'd do this, invalid/non-existing URLs would be fetched again and again with each request. This way `null` (not found) is written to the cache.
  24. Andreas Kienast voted on patch set 16 V +2 CR +2 04/24/23, 08:22

  25. Andreas Kienast merged patch set 17 04/24/23, 08:22