Review main Merged Breaking

65922[!!!][FEATURE] Refactor TER-download handling

Benni Mack CR +2 V +2 Nightmare +1470 −2701 updated 10/09/20, 17:01

Commit message

This change is cleaning up a lot of Extension-Manager-code under-the-hood
in order to solve some important topics related to t3x files, and how
to fetch extensions from TER / Repositories.

First of all, instead of having *one* API on how to retrieve extensions,
an ExtensionDownloaderRemoteInterface is added which allows for downloading
a single extension from a remote endpoint.

In conjunction ListableRemoteInterface is used to fetch all available
extensions from a remote (known as extensions.xml.gz currently).

For both functionalities, the existing functionality from TER API is kept:
* Downloading an extensions.xml.gz
* Downloading an extension as a t3x file from a remote

However, this is now placed into the TerExtensionRemote class.

Custom remote configuration can be added in Services.yaml.

This makes the previous "Repository" and its connected "Mirrors" functionality
completely obsolete.

Additional bugfixes:
* Since TER does not have mirrors anymore, fetching those is not used anymore
* The endpoints now use the correct URL from extensions.typo3.org instead of
repositories.typo3.org (mirrors) and typo3.org (extensions.xml.gz)
* Extensions do not longer depend on a "repository" but on a remote identifier
(such as "ter"). The remotes are configured in Services.yaml and are registered
through a compiler pass, based on the new `extension.remote` tag. This means
that the "tx_extensionmanager_domain_model_repository" DB table is removed,
and therefore does not need to be populated during upgrades / installation
anymore. The Upgrade Wizard is also removed.

In order to ship a new, more flexible download process of extensions,
the TerExtensionRemote can be completely replaced by a third-party
implementation (!) with its custom syntax.

Updating the extension list now directly populates the database,
and uses the BulkExtensionRepositoryWriter instead of the very complex
ExtensionListUtility. This is tightly connected to the TerExtensionRemote
and the XML Parser functionality.

The XML Parser for parsing the mirrors.xml is now removed, and the
XmlParserFactory heavily simplified.

The "Remote" objects now know by themselves when they need to update
and re-import their data. This centralizes the fetching of the remote data,
and also moves the "import into DB" into two separate logic areas.

This cleans up the EM code massively, and reduces the dependency hell of
"utility" and "helper" classes.

In addition, it is now visible how the TerExtensionRemote can be used in
the Upgrade Wizards to download extensions via a cleaner API, which will
become even easier in the future.

Review

  • Code review CR +2 Done
    • Georg Ringer +2
    • Oli Bartsch +1
    • Benni Mack +1
  • Verified V +2 Done
    • Georg Ringer +2
    • Oli Bartsch +1
    • Benni Mack +1
    • TYPO3com +1 CI

Issues 1

Files 52

Show 32 more files

Review log 4176

Show 2751 earlier messages
  1. Benni Mack uploaded patch set 1 09/28/20, 12:54

  2. TYPO3com voted on patch set 1 V -1 09/28/20, 13:08 Completed build in 14m 13s on Mon, 28 Sep, 01:08 PM Test Summary: 126 of 74083 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6041

  3. Benni Mack uploaded patch set 2 09/28/20, 13:13

  4. TYPO3com voted on patch set 2 V -1 09/28/20, 13:14 Completed build in 23s on Mon, 28 Sep, 01:14 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6043

  5. Benni Mack uploaded patch set 3 09/28/20, 14:51

  6. Benni Mack uploaded patch set 4 09/28/20, 15:00

  7. TYPO3com voted on patch set 3 V -1 09/28/20, 15:03 Completed build in 11m 49s on Mon, 28 Sep, 03:03 PM Test Summary: 126 of 74083 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6044

  8. TYPO3com voted on patch set 4 V -1 09/28/20, 15:11 Completed build in 11m 07s on Mon, 28 Sep, 03:11 PM Test Summary: 102 of 74065 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6045

  9. Benni Mack uploaded patch set 5 09/29/20, 08:02

  10. TYPO3com voted on patch set 5 V -1 09/29/20, 08:13 Completed build in 11m 14s on Tue, 29 Sep, 08:13 AM Test Summary: 73582 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6062

  11. Benni Mack uploaded patch set 6 09/29/20, 09:37

  12. TYPO3com voted on patch set 6 V -1 09/29/20, 09:53 Completed build in 15m 33s on Tue, 29 Sep, 09:52 AM Test Summary: 73567 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6063

  13. Benni Mack uploaded patch set 7 09/29/20, 11:19

  14. TYPO3com voted on patch set 7 V -1 09/29/20, 11:33 Completed build in 13m 59s on Tue, 29 Sep, 11:33 AM Test Summary: 73567 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6064

  15. TYPO3com commented on patch set 7 09/29/20, 12:15 Completed build in 1h 10m 00s on Tue, 29 Sep, 12:15 PM Test Summary: 73567 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6064

  16. Benni Mack uploaded patch set 8 09/29/20, 12:56

  17. Benni Mack uploaded patch set 9 09/29/20, 12:57

  18. TYPO3com voted on patch set 8 V -1 09/29/20, 13:13 Completed build in 15m 52s on Tue, 29 Sep, 01:13 PM Test Summary: 73567 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6066

  19. TYPO3com voted on patch set 9 V -1 09/29/20, 13:17 Completed build in 20m 20s on Tue, 29 Sep, 01:17 PM Test Summary: 73567 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6067

  20. Benni Mack commented on patch set 9 09/29/20, 13:28

    Tests are failing because the repository + EM files are instantiated by the Acceptance tests.
  21. Benni Mack uploaded patch set 10 09/30/20, 21:52

  22. Benni Mack updated the commit message in patch set 11 09/30/20, 22:03

  23. TYPO3com voted on patch set 10 V -1 09/30/20, 22:10 Completed build in 18m 01s on Wed, 30 Sep, 10:10 PM Test Summary: 73567 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6107

  24. TYPO3com voted on patch set 11 V -1 09/30/20, 22:27 Completed build in 21m 49s on Wed, 30 Sep, 10:27 PM Test Summary: 73567 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6110

  25. Benni Mack uploaded patch set 12 10/01/20, 06:48

  26. TYPO3com voted on patch set 12 V -1 10/01/20, 07:02 Completed build in 14m 35s on Thu, 1 Oct, 07:02 AM Test Summary: 4 of 74145 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6111

  27. Benni Mack uploaded patch set 13 10/01/20, 07:36

  28. TYPO3com voted on patch set 13 V +1 10/01/20, 07:50 Completed build in 13m 59s on Thu, 1 Oct, 07:50 AM Test Summary: 73660 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6113

  29. Benni Mack updated the commit message in patch set 14 10/01/20, 08:34

  30. TYPO3com voted on patch set 14 V +1 10/01/20, 08:49 Completed build in 14m 15s on Thu, 1 Oct, 08:49 AM Test Summary: 73660 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6115

  31. Benni Mack uploaded patch set 15 10/01/20, 16:03

  32. TYPO3com voted on patch set 15 V -1 10/01/20, 16:15 Completed build in 11m 20s on Thu, 1 Oct, 04:15 PM Test Summary: 3 of 32428 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6127

  33. Georg Ringer commented on patch set 15 10/01/20, 16:26

    RemoteRepository.php:2 Unresolved

    declare_strict

    RemoteNotRegisteredException.php:2 Unresolved

    declare_strict

    Services.php:2 Unresolved

    do we need the license comment here?
  34. Benni Mack uploaded patch set 16 10/01/20, 19:46

  35. TYPO3com voted on patch set 16 V -1 10/01/20, 19:57 Completed build in 11m 06s on Thu, 1 Oct, 07:57 PM Test Summary: 3 of 32428 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6138

  36. Markus Klein updated the commit message in patch set 17 10/01/20, 20:27

  37. TYPO3com voted on patch set 17 V -1 10/01/20, 20:41 Completed build in 13m 59s on Thu, 1 Oct, 08:41 PM Test Summary: 3 of 32428 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6140

  38. Benni Mack uploaded patch set 18 10/02/20, 11:15

  39. TYPO3com voted on patch set 18 V -1 10/02/20, 11:16 Completed build in 25s on Fri, 2 Oct, 11:16 AM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6164

  40. Benni Mack uploaded patch set 19 10/08/20, 07:25

  41. TYPO3com voted on patch set 19 V -1 10/08/20, 07:42 Completed build in 16m 50s on Thu, 8 Oct, 07:42 AM Test Summary: 32354 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6350

  42. Benni Mack uploaded patch set 20 10/08/20, 07:49

  43. TYPO3com voted on patch set 20 V +1 10/08/20, 08:01 Completed build in 11m 31s on Thu, 8 Oct, 08:01 AM Test Summary: 73758 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6352

  44. Georg Ringer voted on patch set 20 CR +1 10/08/20, 08:21

  45. Benni Mack uploaded patch set 21 10/08/20, 09:20

  46. TYPO3com voted on patch set 21 V -1 10/08/20, 09:34 Completed build in 14m 12s on Thu, 8 Oct, 09:34 AM Test Summary: 48 of 74243 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6353

  47. Benni Mack uploaded patch set 22 10/08/20, 10:26

  48. TYPO3com voted on patch set 22 V -1 10/08/20, 10:39 Completed build in 12m 25s on Thu, 8 Oct, 10:39 AM Test Summary: 3 of 74243 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6357

  49. Benni Mack uploaded patch set 23 10/08/20, 11:49

  50. TYPO3com voted on patch set 23 V +1 10/08/20, 12:04 Completed build in 15m 13s on Thu, 8 Oct, 12:04 PM Test Summary: 73734 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6362

  51. Benni Mack uploaded patch set 24 10/08/20, 13:28

  1. TYPO3com voted on patch set 24 V +1 10/08/20, 13:41 Completed build in 13m 21s on Thu, 8 Oct, 01:41 PM Test Summary: 73734 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6367

  2. Oli Bartsch uploaded patch set 25 10/08/20, 16:47

  3. TYPO3com voted on patch set 25 V -1 10/08/20, 17:02 Completed build in 14m 56s on Thu, 8 Oct, 05:02 PM Test Summary: 1 of 74219 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6376

  4. TYPO3com commented on patch set 25 10/08/20, 18:33 Completed build in 2h 00m 04s on Thu, 8 Oct, 06:33 PM Test Summary: 1 of 74219 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6376

  5. Oli Bartsch uploaded patch set 26 10/09/20, 08:48

  6. TYPO3com voted on patch set 26 V -1 10/09/20, 09:06 Completed build in 15m 03s on Fri, 9 Oct, 09:06 AM Test Summary: 72 of 74219 failed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6391

  7. Benni Mack uploaded patch set 27 10/09/20, 10:09

  8. TYPO3com voted on patch set 27 V +1 10/09/20, 10:23 Completed build in 13m 36s on Fri, 9 Oct, 10:23 AM Test Summary: 73734 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6394

  9. Oli Bartsch uploaded patch set 28 10/09/20, 12:30

  10. Benni Mack voted on patch set 28 V +1 CR +1 10/09/20, 12:43

  11. Oli Bartsch uploaded patch set 29 10/09/20, 12:44

  12. TYPO3com voted on patch set 28 V +1 10/09/20, 12:44 Completed build in 14m 05s on Fri, 9 Oct, 12:44 PM Test Summary: 73734 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6400

  13. Oli Bartsch voted on patch set 29 V +1 CR +1 10/09/20, 12:45

  14. TYPO3com voted on patch set 29 V -1 10/09/20, 12:56 Completed build in 12m 06s on Fri, 9 Oct, 12:56 PM Test Summary: 73389 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6401

  15. TYPO3com voted on patch set 29 V +1 10/09/20, 13:47 Completed build in 1h 14m 34s on Fri, 9 Oct, 01:47 PM Test Summary: 73734 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6401

  16. Oli Bartsch uploaded patch set 30 10/09/20, 15:06

  17. TYPO3com voted on patch set 30 V -1 10/09/20, 15:08 Completed build in 31s on Fri, 9 Oct, 03:08 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6408

  18. Oli Bartsch uploaded patch set 31 10/09/20, 15:09

  19. TYPO3com voted on patch set 31 V -1 10/09/20, 15:09 Completed build in 25s on Fri, 9 Oct, 03:09 PM Test Summary: No tests found Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6409

  20. Oli Bartsch uploaded patch set 32 10/09/20, 15:11

  21. Oli Bartsch voted on patch set 32 V +1 CR +1 10/09/20, 15:21

  22. TYPO3com voted on patch set 32 V +1 10/09/20, 15:25 Completed build in 14m 15s on Fri, 9 Oct, 03:25 PM Test Summary: 73733 passed Find logs and detail information at https://bamboo.typo3.com/browse/CORE-GTC-6410

  23. Benni Mack voted on patch set 32 V +1 CR +1 10/09/20, 15:34

  24. Georg Ringer voted on patch set 32 V +2 CR +2 10/09/20, 17:01

  25. Georg Ringer merged patch set 33 10/09/20, 17:01