Issue Bug New Orphaned

#106585database:updateschema breaks on Data Type 'VECTOR'

Nikolai Townsend 04/17/25, 14:26 updated 05/23/25, 21:21

Description

When running vendor/bin/typo3 database:updateschema on CLI or “Analyze Database Structure” in the Install Tool, they throw an error.

This happens when you got a ext_tables.sql in your project that adds a field with the new Data Type ‘VECTOR’.

myext/ext_tables.sql

CREATE TABLE tx_myext_domain_model_chunk
(
    pid       tinyint(3) unsigned DEFAULT '0' NOT NULL,
    title     VARCHAR(255) NOT NULL DEFAULT '',
    url       VARCHAR(255) NOT NULL DEFAULT '',
    content   LONGTEXT     NOT NULL DEFAULT '',
    embedding VECTOR(1024) NOT NULL,
    VECTOR    INDEX (embedding)
);

This new data type was introduced on in MariaDB 11.7.2: https://mariadb.com/kb/en/vector-overview/

This data type is used in AI for search of text by their coordinates in an multi dimensional space.

Database analysis failed
[SQL Error] line 0, col 240: Error: Expected BIT, TINYINT, SMALLINT, MEDIUMINT, INT, INTEGER, BIGINT, REAL, DOUBLE, FLOAT, DECIMAL, NUMERIC, DATE, TIME, TIMESTAMP, DATETIME, YEAR, CHAR, VARCHAR, BINARY, VARBINARY, TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB, TINYTEXT, TEXT, MEDIUMTEXT, LONGTEXT, ENUM, SET, or JSON, got 'VECTOR' in statement: CREATE TABLE tx_nitosite_domain_model_chunk ( pid tinyint(3) unsigned DEFAULT '0' NOT NULL, title VARCHAR(255) NOT NULL DEFAULT '', url VARCHAR(255) NOT NULL DEFAULT '', content LONGTEXT NOT NULL DEFAULT '', embedding VECTOR(1024) NOT NULL, VECTOR INDEX (embedding) );

When I add this field manually to the DB, TYPO3 is working fine, I can even add vectors to this table with doctrine.

When I add it manually and remove it from ext_tables.sql it still crashes, just with stack trace:

(1/1) Doctrine\DBAL\Exception\InvalidArgumentException
Unknown database type "vector" requested, TYPO3\CMS\Core\Database\Platform\MariaDB1010Platform may not support it.

in /var/www/html/vendor/doctrine/dbal/src/Platforms/AbstractPlatform.php line 354

        $dbType = strtolower($dbType);

        if (! isset($this->doctrineTypeMapping[$dbType])) {
            throw new InvalidArgumentException(sprintf(
                'Unknown database type "%s" requested, %s may not support it.',
                $dbType,
                static::class,
            ));
at Doctrine\DBAL\Platforms\AbstractPlatform->getDoctrineTypeMapping('vector')
in /var/www/html/vendor/typo3/cms-core/Classes/Database/Schema/SchemaManager/ColumnTypeCommentMethodsTrait.php line 57
     */
    private function determineColumnType(string $dbType, array &$tableColumn): string
    {
        $platform = ($this instanceof AbstractPlatform) ? $this : $this->platform;
        $type = $dbType !== '' ? $platform->getDoctrineTypeMapping($dbType) : '';
        if (isset($tableColumn['comment'])) {
            $type = $this->extractDoctrineTypeFromComment($tableColumn['comment'], $type);
            $tableColumn['comment'] = $this->removeDoctrineTypeFromComment($tableColumn['comment'], $type);
        }
at TYPO3\CMS\Core\Database\Schema\SchemaManager\MySQLSchemaManager->determineColumnType('vector', array('table_name' => 'tx_nitosite_domain_model_chunk', 'field' => 'embedding', 'type' => 'vector(1024)', 'null' => 'NO', 'key' => 'MUL', 'default' => null, 'extra' => '', 'comment' => '', 'characterset' => null, 'collation' => null, 'name' => ''))
in /var/www/html/vendor/typo3/cms-core/Classes/Database/Schema/SchemaManager/MySQLSchemaManager.php line 241
        // Following line differs from \Doctrine\DBAL\Schema\MySQLSchemaManager::_getPortableTableColumnDefinition,
        // taken from:
        // - https://github.com/doctrine/dbal/blob/61446f07fcb522414d6cfd8b1c3e5f9e18c579ba/src/Schema/MySQLSchemaManager.php#L186-L192
        // - https://github.com/doctrine/dbal/blob/61446f07fcb522414d6cfd8b1c3e5f9e18c579ba/src/Schema/PostgreSQLSchemaManager.php#L427-L429
        $type = $this->determineColumnType($dbType, $tableColumn);

        switch ($dbType) {
            case 'char':
            case 'binary':
at TYPO3\CMS\Core\Database\Schema\SchemaManager\MySQLSchemaManager->parentGetPortableTableColumnDefinition(array('table_name' => 'tx_nitosite_domain_model_chunk', 'field' => 'embedding', 'type' => 'vector(1024)', 'null' => 'NO', 'key' => 'MUL', 'default' => null, 'extra' => '', 'comment' => '', 'characterset' => null, 'collation' => null, 'name' => ''))
in /var/www/html/vendor/typo3/cms-core/Classes/Database/Schema/SchemaManager/MySQLSchemaManager.php line 108
        /** @var DoctrineMariaDBPlatform|DoctrineMySQLPlatform $platform */
        $platform = $this->platform;
        $tableColumn = $this->normalizeTableColumnData($tableColumn, $platform);
        return $this->processCustomDoctrineTypesColumnDefinition($tableColumn, $platform)
            ?? $this->parentGetPortableTableColumnDefinition($tableColumn);
    }

    /**
     * @param array<string, mixed> $tableColumn
at TYPO3\CMS\Core\Database\Schema\SchemaManager\MySQLSchemaManager->_getPortableTableColumnDefinition(array('TABLE_NAME' => 'tx_nitosite_domain_model_chunk', 'field' => 'embedding', 'type' => 'vector(1024)', 'null' => 'NO', 'key' => 'MUL', 'default' => null, 'EXTRA' => '', 'comment' => '', 'characterset' => null, 'collation' => null))
in /var/www/html/vendor/doctrine/dbal/src/Schema/AbstractSchemaManager.php line 670
    protected function _getPortableTableColumnList(string $table, string $database, array $tableColumns): array
    {
        $list = [];
        foreach ($tableColumns as $tableColumn) {
            $column = $this->_getPortableTableColumnDefinition($tableColumn);

            $name        = strtolower($column->getQuotedName($this->platform));
            $list[$name] = $column;
        }

The people from DDEV said, they are planing to add MariaDB 11.8 when the LTS is released.

Beside the DB compare, TYPO3 12 + 13 seem to work well with MariaDB 11.7.x.

Reviews 0

No change resolves or names this issue in its commit message, and no comment links one.

Fix this issue

Start the commit message of your patch with this. Add what the patch does, and the versions it goes to in "Releases:". How to create a patch

Commit message
[BUGFIX] database:updateschema breaks on Data Type 'VECTOR' Resolves: #106585 Releases: main

History 6

  1. Stefan Bürk commented and changed assigned to Stefan Bürk <span data-processor="relativetime" data-value="2025-04-17T17:01:49+00:00" > 04/17/25, 17:01 </span>

    @nito Thanks for the report.

    The first direct thing is, that TYPO3 has no official support for MariaDB 11.0 or newer
    and our core tests are failing with these newer versions which has to be tackled first.
    At least for 13.4 and main. We will not announce official support for v12 as long as we
    cannot do that for 13.4 or main (v14).

    The second thing here is, that ext_tables.sql provides only a defined subset of MySQL
    DDL to allow TYPO3 to ensure compatibility across all supported database systems and
    versions with the same behaviour.

    I need to check the new field type and how a full compat can be archieved. In general,
    that includes extending the TYPO3 parser for it and match it to a Doctrine DBAL Type,
    which is a first line if Doctrine DBAL has full support for this new field accross all
    Database Systems in the 4.2 (or upcoming 4.3.0) version.

    TYPO3 v12 is in priority bugfix mode, so don’t assume that this will happen in TYPO3 v12,
    even v13 is most likely not a candidate for it. When possible, it may be a option for v13.
    But all that is not written in stone, but you should be aware of that.

    This is not a bug, it’s a feature request as the required MariaDB version is not officially
    supported nor did TYPO3 ever annonced to support all database field types in ext_tables.sql.
    I keep it for now as “bug” report, but this may be changed when investigating this in more
    detail.

  2. Stefan Bürk commented <span data-processor="relativetime" data-value="2025-05-23T15:01:03+00:00" > 05/23/25, 15:01 </span>

    It seams that doctrine itself does not have support for VECTOR yet.

    https://github.com/doctrine/dbal/issues/6703

    We won’t add anythin in that area as long as doctrine does not support it directly.

  3. Nikolai Townsend commented <span data-processor="relativetime" data-value="2025-05-23T21:21:11+00:00" > 05/23/25, 21:21 </span>

    Stefan Bürk wrote in #note-2:
    > It seams that doctrine itself does not have support for VECTOR yet.
    >
    > https://github.com/doctrine/dbal/issues/6703
    >
    > We won’t add anythin in that area as long as doctrine does not support it directly.

    In a raw way it is actually possible:

            $connection->executeStatement(
                'INSERT INTO tx_myext_domain_model_chunk (pid, title, url, content, embedding)
                            VALUES (:pid, :title, :url, :content, VEC_FromText(:embedding))',
                [
                    'pid' => $pid,
                    'title' => $title,
                    'url' => $url,
                    'content' => $content,
                    'embedding' => $embedding,
                ]
            );

    There is no way to handle VEC_FromText, VEC_ToText, VEC_DISTANCE* in an object oriented way with methods etc.

    As a workaround, to get the ability to do a db-compare back, I build me a command that:

    • backup my tx_myext_domain_model_chunk
    • delete tx_myext_domain_model_chunk
    • run database:updateschema
    • restore tx_myext_domain_model_chunk

    Not the nicest solution, but it might do the job for a while.