Review main Abandoned

94961[TASK] Store remote extension list as JSON files

Benni Mack CR 0 V 0 Expert +1735 −642 updated 09/11/26, 21:43

Commit message

The extension manager kept its local mirror of the packages available in
TER in the database table tx_extensionmanager_domain_model_extension:
every update of the extension list deleted and bulk-inserted tens of
thousands of rows, the table easily grew beyond 100 MB, and every lookup
went through SQL again although the data is a plain, read-only cache of
the downloaded extensions.xml.gz file.

The downloaded list is now converted into JSON files below
var/extensionmanager/<remote>/. ExtensionRepository keeps its query API
but serves all lookups from that store, so the table is neither read nor
written anymore. An already downloaded extensions.xml.gz is converted
automatically the next time the list is accessed, which keeps existing
installations working without manual interaction.

The list is stored as the views that are actually queried rather than as
one file holding everything: current.json for browsing and searching,
distributions.json, and versions/<extensionkey>.json for the lookups
about a single package. Nothing reads the whole list of a remote, and
the import writes a package as soon as its last version was parsed, so
neither reading nor importing depends on how large the list is. Against
the TER list of about 28.000 package versions, counting the available
extensions reads 2 MB instead of 86 MB, resolving one package key a few
kilobytes, and importing peaks at 6 MB instead of 142 MB.

Which store a remote uses is resolved by ExtensionListStoreRegistry, so
a remote whose list does not fit the file based store can register its
own implementation of ExtensionListStoreInterface. Parsing the
downloaded XML moved out of the store into ExtensionListImporter, which
feeds the parsed versions into whichever store the remote uses.
BulkExtensionRepositoryWriter is superseded by all of this and removed.

Since entries of a remote list have no local database identity, the uid
property of the Extension model is dropped. Addressing a package by key,
version and remote has been prepared with #110304, so no call site needs
to change for that.

This also allows to fix the flaw that led to the revert of #110144 at
its source: Extension::createFromExtensionArray() no longer binds the
created object to the "ter" remote. An extension object created from a
locally available extension now always carries an empty remote, and
ExtensionManagementService::downloadMainExtension() - which can no
longer look at a uid - only fetches packages that were actually resolved
from a remote listing. Local extensions, whether released to TER or not,
are therefore never overwritten by a forced download when they are
installed via the extension manager.

The now unused database table and its schema definition are removed in a
follow-up.

Review

  • Code review CR 0
  • Verified V 0
    • core-ci +1 CI

Issues 4

Files 21

Show 1 more file

Review log 1117

  1. Benni Mack uploaded patch set 1 07/27/26, 15:39

  2. core-ci voted on patch set 1 V +1 07/27/26, 15:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105478

  3. Benni Mack uploaded patch set 2 07/27/26, 16:13

    Copied Votes: * Verified+1 (copy condition: "**changekind:NO_CHANGE**")
  4. core-ci voted on patch set 2 V -1 07/27/26, 16:27 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105484

  5. Andreas Kienast commented on patch set 2 07/28/26, 06:14

    RemoteExtensionListStore.php:41

    ```suggestion private const string FILE_SUFFIX = '.extensions.json'; ```

    RemoteExtensionListStore.php:52

    ```suggestion private const string MINIMUM_DATE_TO_IMPORT = '2017-04-04T00:00:00+00:00'; ```

    RemoteExtensionListStore.php:118

    `array_merge()` per row gets expensive over time. What about ``` $rows = [[]]; foreach (...) { $rows[] = $this->getByRemoteIdentifier($remoteIdentifier); } return array_merge(...$rows); ```
  6. Benni Mack uploaded patch set 3 V -1 07/30/26, 18:00

  7. core-ci voted on patch set 3 V -1 07/30/26, 18:13 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105709

  8. Benni Mack rebased onto patch set 4 V -1 07/31/26, 16:23

  9. core-ci voted on patch set 4 V -1 07/31/26, 16:36 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105756

  10. Benni Mack uploaded patch set 5 V -1 08/01/26, 06:32

  11. core-ci voted on patch set 5 V +1 08/01/26, 06:40 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/105780

  12. Benni Mack commented on patch set 5 08/03/26, 10:05

    RemoteExtensionListStore.php:41 Reply

    Done

    RemoteExtensionListStore.php:52 Reply

    Done

    RemoteExtensionListStore.php:118 Reply

    Done
  13. Benni Mack marked it ready for review 08/06/26, 18:23

  14. Oliver Klee voted on patch set 5 V +1 08/06/26, 21:56

  15. Benni Mack uploaded patch set 6 V +1 08/14/26, 20:21

  16. core-ci voted on patch set 6 V +1 08/14/26, 20:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106717

  17. Benni Mack abandoned the change 09/11/26, 21:43

    wont do for now