#110420Rework the database analyzer stack
Description
Problem
The database analyzer describes a schema change as an SQL statement, grouped by the kind of operation and keyed by the md5 hash of the statement itself. What a change does, whether it removes anything, and which statements belong together can only be re-derived from those array keys, and every consumer does that on its own – usually with a hardcoded list such as ['add', 'change', 'create_table', 'change_table'].
Statements of one table depend on each other, so offering each of them for selection allows a table to be left in a state it cannot be left in: an index can be selected without the column it needs, and on a platform that rebuilds a table to alter it, half of a copy, drop, create and copy-back sequence can be applied.
On top of that the stack carries mutable copies of doctrine classes that doctrine marks as final (TableDiff, SchemaDiff, Comparator), extends the doctrine platforms for a type comment that only MySQL needs, and introspects through an API doctrine has deprecated.
Scope
This epic collects the rework: the table becomes the unit of analysis and application, the result becomes a set of immutable objects, the doctrine extensions go away, and the install tool and the command line are built on the new API. It also collects the bugs found while doing it – two of which can destroy a database.
The whole analyzer stack is annotated internal@ and is not part of the public core API. The changes are still documented, because extensions do reach into it.
Sub-tasks
The sub-tasks are listed below and are meant to be reviewed and merged in the order of the Gerrit relation chain:
- #110421 – Install tool answers with an empty response when an exception is handled – Gerrit 95186
- #110422 – An index over a column that does not exist destroys a SQLite database – Gerrit 95187
- #110423 –
ALTER TABLE ADD COLUMNfails on SQLite for json and uuid columns – Gerrit 95188 - #110424 – Introspect the schema through the non-deprecated doctrine API – Gerrit 95189
- #110425 – Reduce column type comment handling to MySQL – Gerrit 95190
- #110426 – Generate table option changes without platform hooks – Gerrit 95191
- #110427 – Rework the database analyzer stack onto objects – Gerrit 95192
- #110428 – Console command for database schema updates – Gerrit 95193
- #110429 – Remove the array shaped schema migrator API – Gerrit 95194
- #110430 – Remove
SchemaMigrator::install()– Gerrit 95195
Between sub-task 3 and sub-task 4 the relation chain carries one more change that is not a sub-task of this epic – see the next section.
The one change of the series that gets no sub-task
Stack position 4, Gerrit 95240, [BUGFIX] Ignore tables that exist as views, Releases: main, 14.3, 13.4.
It resolves the already existing, already reported bug #106546 – TYPO3 Database Analyzer should ignore tables that exist as views – and carries Resolves: #106546 in its commit message. No new issue is created for it.
It sits below the rework on purpose and is implemented in the old ConnectionMigrator, which is what makes it backportable to 14.3 and 13.4. The rework (sub-task 7, Gerrit 95192) carries the same behaviour into DatabaseSchemaAnalyzer as part of itself, with no separate mention, and sub-task 9 (Gerrit 95194) deletes the old path together with its test.
That change takes views out of the comparison and nothing else. Making what is compared configurable in general belongs to #101235 and is not claimed here.
Not in scope, on purpose
- A SQLite database that already contains an index over a missing column cannot be repaired by TYPO3. The
TypeErrorcomes out of the doctrine generator before TYPO3 sees anything, and the metadata provider cannot be decorated – platforms narrow the return type ofcreateMetadataProvider()to final readonly classes. Sub-task 2 prevents the state; an existing one needs a manualDROP INDEX. - Making the comparison scope configurable in general – which tables and which connections are compared at all – belongs to #101235 and is not part of this series. The view fix at stack position 4 is the view case only.
- The extension point for
ColumnDataMigration. The rework prepares for it and marks the places with @todo markers, but whether it becomes a PSR-14 event or an interface with registered migrations is deliberately not decided in this series. See sub-task 7.
Patches
All changes are pushed to Gerrit under the topic database-analyzer.
Related, not claimed
#101235, #93643, #106719, #108028.
Reviews 23
- main 95987[FEATURE] Add connection option ignoreTables
- main 95195[WIP][!!!][TASK] Remove SchemaMigrator::install()
- main 95194[WIP][!!!][TASK] Remove the array shaped schema migrator API
- main 95193[WIP][FEATURE] Add database schema update command
- main 95192[WIP][TASK] Rework database analyzer stack
- main 95191[WIP][TASK] Generate table option changes without platform hooks
- main 95190[WIP][TASK] Reduce column type comment handling to MySQL
- main 95189[WIP][TASK] Use non-deprecated doctrine introspection API
- 13.4 95561[TASK] Raise doctrine/dbal to ~4.4.4
- 14.3 95560[TASK] Raise doctrine/dbal to ~4.4.4
- main 95263[TASK] Raise doctrine/dbal to ~4.4.4
- 13.4 95554[BUGFIX] Add SQLite JSON and UUID column declarations
- 14.3 95553[BUGFIX] Add SQLite JSON and UUID column declarations
- main 95188[BUGFIX] Add SQLite JSON and UUID column declarations
- 13.4 95552[BUGFIX] Reject indexes over undeclared columns
- 14.3 95551[BUGFIX] Reject indexes over undeclared columns
- main 95187[BUGFIX] Reject indexes over undeclared columns
- 13.4 95487[BUGFIX] Provide error page to failsafe container
- 14.3 95486[BUGFIX] Provide error page to failsafe container
- main 95186[BUGFIX] Provide error page to failsafe container
- main 95240[BUGFIX] Ignore tables that exist as views
- 14.3 95390[BUGFIX] Ignore tables that exist as views
- 13.4 95391[BUGFIX] Ignore tables that exist as views
Subtasks 11
- Bug #110421Install tool answers with an empty response when an exception is handled
- Bug #110422An index over a column that does not exist destroys a SQLite database
- Bug #110423ALTER TABLE ADD COLUMN fails on SQLite for json and uuid columns
- Task #110424Introspect the schema through the non-deprecated doctrine API
- Task #110425Reduce column type comment handling to MySQL
- Task #110426Generate table option changes without platform hooks
- Task #110427Rework the database analyzer stack onto objects
- Feature #110428Console command for database schema updates
- Task #110429Remove the array shaped schema migrator API
- Task #110430Remove SchemaMigrator::install()
- Task #110438Raise doctrine/dbal to ~4.4.4 and support the MariaDB 12.3 platform
Related issues 12
- Bug #93643Database Analyzer Length of Double not synchronized
- Task #101235Make additional DB connections optional sometimes, e.g. in case of SchemaMigration
- Bug #103141TCA type=uuid does not use native uuid type for postgresql
- Feature #103154Ignore tables during database analysis
- Bug #105310\TYPO3\CMS\Core\Database\Schema\Parser\Parser translates CHAR to VARCHAR
- Bug #106546TYPO3 Database Analyzer should ignore tables that exist as views.
- Bug #106719Install tool - SchemaMigrator & DBAL 4 - MySQL float type
- Bug #106803Doctrine DBAL 4.3-dev exception on "InvalidColumnDeclaration" in DB compare (EXT:seo pages.sitemap_priority column)
- Bug #108028Database-Compare fails on database-names like "a.b.c"
- Bug #108051Allow `UUID` in `ext_tables.sql` using Doctrine `GuidType`
- Epic #110253Prepare database abstraction layer for Doctrine DBAL 5.x
- Feature #110761Let the Database Analyzer ignore configured tables on the Default connection