Merge conflict Gerrit cannot merge this change. It needs a rebase onto main.

Review main Work in progress Stale

95192[WIP][TASK] Rework database analyzer stack

Stefan Bürk CR 0 V 0 Nightmare +5239 −372 updated 09/02/26, 13:35

Commit message

The database analyzer described a schema change as an SQL statement,
grouped by the kind of operation and keyed by the md5 hash of the
statement itself. What a change does, whether it removes anything,
and which statements belong together could only be re-derived from
those array keys, and every consumer did that on its own - usually
with a hardcoded list such as
`['add', 'change', 'create_table', 'change_table']`.

Statements of one table depend on each other, so offering each of
them for selection allowed a table to be left in a state it cannot be
left in: an index could be selected without the column it needs, and
on a platform that rebuilds a table to alter it, half of a copy,
drop, create and copy-back sequence could be applied.

The stack is reworked around objects, and the table becomes the unit
of both analysis and application.

The comparison is built in one pass and returned as an immutable
`Dto\SchemaComparison`, instead of being assembled by rewriting a
doctrine `SchemaDiff` in place. The doctrine diff is used as it is,
and what TYPO3 needs on top of it lives next to it: `TableRename`
names the target of a table renamed to the deletion prefix, which
doctrine has no notion of, and the changed table options, which
doctrine neither compares nor applies, are a map next to the altered
tables. The three in place `migrate*()` methods collapse into one
`rebuildTableDiff()`, which undoes the rename doctrine guesses for an
added and a dropped column that only differ by name - with partial
`ext_tables.sql` files that guess is wrong far more often than it is
right - and turns a dropped column without the deletion prefix into a
rename to it.

From that comparison `DatabaseSchemaAnalyzer` builds change sets:

* `ChangeType` and `ChangeSafety` - the kind of change, and the
  single source of truth for the safe/destructive classification,
  replacing the hardcoded operation key lists.
* `RemovalMode` - the two steps a removal takes. What is no longer
  expected is first renamed to the `zzz_deleted_` prefix, which keeps
  its data, and only what already carries that prefix is dropped. The
  two never describe the same object, so they compose instead of
  excluding each other.
* `Dto\SchemaChange` - one change with the statements that make it
  up. A change carries a list, because a platform that cannot alter a
  table in place expresses one change as a copy, drop, create and
  copy-back sequence.
* `Dto\TableChangeSet` - all changes of one table that are applied
  together. It enforces in its constructor that a set is either
  entirely safe or entirely destructive.
* `Dto\SchemaChangeSet` - the change sets across all connections,
  with the filters consumers would otherwise write themselves.
* `Dto\SchemaOperation` - what a change set does, described rather
  than rendered, for consumers that need the structure instead of the
  SQL.
* `Dto\ApplyResult` - what was applied and what failed, with the
  table each failure belongs to.

`SchemaMigrator` gains the counterpart: `analyze()` returns the
change sets, `apply()` applies the ones the given hashes select, and
`applySafe()` applies everything that removes nothing. A change set is
applied as one unit - if a statement fails, the remaining statements
of that set are skipped and the set is reported as failed, instead of
being left half applied.

The security measure of the install tool is kept and now holds by
construction rather than by convention: `apply()` re-derives the
change sets and only intersects them with the given hashes, so
statements are never taken from a request. A hash the analysis did not
produce selects nothing.

How many statements a change set needs is up to the platform, not up
to the analyzer. All changes of a table are rendered in one call, and
`AlterTableStatementAggregator` folds the result into a single
`ALTER TABLE` wherever the platform accepts a list of actions. MySQL
and MariaDB already render one combined statement, PostgreSQL renders
one per action and is folded afterwards, and SQLite accepts exactly
one action per `ALTER TABLE` and cannot be folded at all.

Two column changes need more than the definition change doctrine
produces for them, and both lose data if those statements are not
executed together with the change. `ColumnDataMigration` carries
them:

* The hash columns of EXT:indexed_search changed from an integer to a
  string. The stored values cannot be converted and altering the
  column fails on the existing rows, so the table is emptied first.
* doctrine/dbal 4 implements auto increment columns on PostgreSQL as
  identity columns instead of as a serial. Letting it drop the old
  sequence and create an identity column throws away the current
  sequence value, so the sequence is converted in place.

Both are wired into that class directly for now, and the class is
shaped for taking them out again rather than for keeping them. It is a
stateless service, `process()` takes a `Dto\ColumnDataMigrationRequest`
and returns a `Dto\ColumnDataMigrationResult`, and the request already
carries more than the two implementations read: the connection name
next to the connection, and a `ColumnDataMigrationStage` naming which
part of a change is being rendered. Both exist so that whatever asks
the migrations later - a PSR-14 event, or registered migrations behind
an interface - has what it needs without changing this shape again.
Which of the two it becomes is not decided here.

The call sites are prepared the same way: renaming and dropping whole
tables ask as well, although nothing reacts to those stages today. A
stage nothing matches returns its statements unchanged, which is what
makes each of them a place where something can be plugged in without
moving the call. The places where an event would be dispatched carry
`@todo` markers.

The consumer builds the service with `GeneralUtility::makeInstance()`
instead of having it injected. It has no dependency of its own yet, so
there is nothing to inject; that changes together with the step above.

The install tool follows the same unit: one card per table, headed by
what happens to the table itself, and a small table per kind of change
below it - added fields, modified fields, added indexes, modified
indexes - each shown only where there is something to show. A change
carries both declarations, because knowing only the new one says
nothing about whether the change is wanted. A card has two tabs, the
changes and the statements they render into, so nothing is a control
that reveals something once and is then spent. The statements are one
block to copy out, terminated for the reader; the hash is built from
the statements as the analysis produced them, so that terminator never
reaches what is sent back.

The statements of everything ticked are shown together for the whole
step, and what a step executed can be shown after it ran. The latter
is off by default, since it is a review aid rather than part of
getting through the wizard.

Copying reports next to the control that was used instead of through a
notification, which renders behind the modal backdrop and is gone
before it can be read.

Operations carry their type next to their label, so the display no
longer has to recognise English wording to know what it is showing.
The mapping from an analysis to what is displayed moved into
`DatabaseAnalyzerSuggestionService`, so the structure the JavaScript
reads is covered by a test built from a real analysis.

The groups are offered one at a time. The module walks them as a
wizard, re-analyzing the schema after every applied step, so the next
step is built from what the previous one left rather than from the
analysis the module started with. The steps are whatever the analysis
returns; a step with nothing in it is neither shown nor counted.

This is deliberately a first iteration, and it is expected to change
further as upcoming features and fixes land. A full redesign is on the
table, either together with UX experts once the feature set has
settled, or as part of the install tool modal rework. That has been
discussed and is not decided yet.

`PackageSetup` uses `applySafe()` and no longer computes the whole
comparison twice just to apply it.
`DatabaseUpgradeWizardsService` reads the detected operations instead
of walking raw doctrine diffs, which is what
`SchemaMigrator::getSchemaDiffs()` existed for - it had no other
consumer and is removed.

The array shaped suggestion API keeps building its own comparison for
now, together with the mutable diff copies it needs. Both are removed
in a later change of this series.

Everything in the analyzer stack is annotated `@internal` and is not
part of the public core API.

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 13
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/92/95192/13 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/92/95192/13 && git cherry-pick FETCH_HEAD

Review

  • Code review CR 0 Needs a vote
  • Verified V 0 Needs a vote
    • core-ci +1 CI

Relation chain 8

  1. main 95195[WIP][!!!][TASK] Remove SchemaMigrator::install() Based on this change Work in progress Merge conflict Stefan Bürk CR 0 V 0 database-analyzer Normal +62 −37
  2. main 95194[WIP][!!!][TASK] Remove the array shaped schema migrator API Based on this change Work in progress Merge conflict Stefan Bürk CR 0 V 0 database-analyzer Nightmare +1441 −3637
  3. main 95193[WIP][FEATURE] Add database schema update command Based on this change Work in progress Stefan Bürk CR 0 V 0 database-analyzer Expert +1066 −0
  4. 95192 [WIP][TASK] Rework database analyzer stack This change

  5. main 95191[WIP][TASK] Generate table option changes without platform hooks This change is based on it Work in progress Stefan Bürk CR 0 V 0 database-analyzer Hard +92 −282
  6. main 95190[WIP][TASK] Reduce column type comment handling to MySQL This change is based on it Work in progress Stefan Bürk CR 0 V 0 database-analyzer Normal +50 −105
  7. main 95189[WIP][TASK] Use non-deprecated doctrine introspection API This change is based on it Work in progress Stefan Bürk CR 0 V 0 → 14.3, 13.4 database-analyzer Expert +450 −599
  8. main 95263[TASK] Raise doctrine/dbal to ~4.4.4 This change is based on it Merged Stefan Bürk CR +2 V +2 → 14.3, 13.4 database-analyzer Normal +57 −5

Issues 2

Files 48

Show 28 more files

Review log 1326

Show 1 earlier messages
  1. Stefan Bürk uploaded patch set 1 08/09/26, 21:00

  1. core-ci voted on patch set 1 V -1 08/09/26, 21:23 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106232

  2. Stefan Bürk uploaded patch set 2 V -1 08/09/26, 22:35

  3. core-ci voted on patch set 2 V +1 08/09/26, 22:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106236

  4. Stefan Bürk uploaded patch set 3 V +1 08/11/26, 00:32

  5. core-ci voted on patch set 3 V +1 08/11/26, 00:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106356

  6. Stefan Bürk uploaded patch set 4 V +1 08/11/26, 04:00

  7. core-ci voted on patch set 4 V +1 08/11/26, 04:21 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106371

  8. Stefan Bürk rebased onto patch set 5 V +1 08/11/26, 05:17

  9. core-ci voted on patch set 5 V +1 08/11/26, 05:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106390

  10. Stefan Bürk uploaded patch set 6 V +1 08/11/26, 11:38

  11. core-ci voted on patch set 6 V +1 08/11/26, 11:58 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106421

  12. Stefan Bürk rebased onto patch set 7 V +1 08/11/26, 13:29

  13. core-ci voted on patch set 7 V +1 08/11/26, 13:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106437

  14. Stefan Bürk uploaded patch set 8 08/11/26, 14:14

    Copied Votes: * Verified+1 (copy condition: "**changekind:NO_CHANGE**")
  15. core-ci voted on patch set 8 V +1 08/11/26, 14:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106451

  16. Stefan Bürk rebased onto patch set 9 V +1 08/14/26, 20:48

  17. core-ci voted on patch set 9 V +1 08/14/26, 21:13 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106727

  18. Stefan Bürk rebased onto patch set 10 V +1 08/31/26, 10:28

  19. core-ci voted on patch set 10 V -1 08/31/26, 11:04 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107411

  20. Stefan Bürk rebased onto patch set 11 V -1 09/02/26, 10:40

  21. core-ci voted on patch set 11 V -1 09/02/26, 11:17 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107596

  22. Stefan Bürk uploaded patch set 12 V -1 09/02/26, 11:40

  23. core-ci voted on patch set 12 V +1 09/02/26, 11:51 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107612

  24. Stefan Bürk rebased onto patch set 13 V +1 09/02/26, 13:16

  25. core-ci voted on patch set 13 V +1 09/02/26, 13:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107631