Merge conflict Gerrit cannot merge this change. It needs a rebase onto main.

Review main Work in progress Orphaned

87719[WIP][BUGFIX] Mitigate missing column in cached schema information

Stefan Bürk CR 0 V 0 Easy +50 −0 updated 07/23/25, 14:11

Commit message

Introducing native database JSON field support (#99062) enforced
cacheing schema information for TYPO3 managed connections due to
performance reasons. `TCA type=json` support (#99226) was added
on top intoducing an addition value check in TYPO3's DataHandler.

TYPO3 API like `DataHandler` requires TYPO3 managed databases and
connection and are not suitable for non-managed connections.

The TYPO3 database analyzer has been adjusted to properly clear
connection schema information cache after processing changes to
ensure non-stalled information. `DataHandler` changes requiring
schema information has been added assuming non-stalled data but
not tightened against stalled schema information, which happens
when integrator or developers making external database structure
changes for managed connections and misses to flush related db
schema cache state.

This change mitigates stalled (non-existing) column information
in `DataHandler::compareFieldArrayWithCurrentAndUnset()` omiting
detecting column type to avoid Doctrine DBAL Schema Exception is
thrown and not handled by simply passing the value as is, which
releaxes the situation a little bit. Note database/query errors
can still occur for some database backends when the value is not
correctly types and not reliable to mitigate further. Flushing
the cache properly would correctly fix the issue.

To communicate this more transparantly, a new important changelog
entry is added now next to the related feature changelog entry for
the native JSON fieldtype to emphasize the need to use either the
Database Analyzer or ensure to flush caches after external schema
changes for managed connections.

Test this change

Run a command in your clone of the TYPO3 Core. Then try what the change does and vote on Verified in Gerrit. How to review a patch

Check out patch set 5
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/19/87719/5 && git checkout FETCH_HEAD
Cherry-pick onto main
git fetch https://review.typo3.org/Packages/TYPO3.CMS refs/changes/19/87719/5 && git cherry-pick FETCH_HEAD

Review

  • Code review CR 0 Needs a vote
  • Verified V 0 Needs a vote
    • core-ci +1 CI

Issues 3

Files 2

Review log 911

  1. Stefan Bürk uploaded patch set 1 01/13/25, 14:45

  2. Stefan Bürk updated the commit message in patch set 2 01/13/25, 14:46

  3. Stefan Bürk updated the commit message in patch set 3 01/13/25, 14:47

  4. Stefan Bürk updated the commit message in patch set 4 01/13/25, 14:48

  5. core-ci voted on patch set 4 V +1 01/13/25, 14:56 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/72361

  6. Stefan Bürk updated the commit message in patch set 5 V +1 01/13/25, 16:36

  7. Christian Kuhn commented on patch set 4 CR -1 01/13/25, 16:37

    naughty method. it shouldn't exist at all. i'll try to reflect on this change a bit to come to a conclusion on what to do in which versions.
  8. core-ci voted on patch set 5 V +1 01/13/25, 16:44 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/72369

  9. Stefan Bürk wrote 04/03/25, 08:52

    Hashtag added: lolli-feedback
  10. Benjamin Franzke commented on patch set 5 04/03/25, 19:18

    Unresolved

    I think I already proposed this in the past, so sorry to repeat, but can't we fix this by relying on the schema API instead of introspecting the database at all. Yes, that would break when the database is not up2date with schema API, but that's actually what I'd expect, if a databse field is outdated, it does not work. By introspecting the database we basically build a runtime workaround for non-updated database schematas, instead of reyling on our schema and expect it to be applied.

    Important-99062-CachedDatabaseSchemaInformationForTYPO3ManagedConnections.rst:35 Unresolved

    It is: `cache:flush -g system`

    Important-99062-CachedDatabaseSchemaInformationForTYPO3ManagedConnections.rst:39 Unresolved

    This message is hard to grasp. What does this mean for the integrator/developer? Does it mean that it is disallowed to query a database via ConnectionPool that was not created via TYPO3 schema manager? Is this relevant for *some* types or all? I'm sure there are a lot instances out there that use the ConnectionPool to connect to some external databases, e.g., to import data or simply to use it directly. This would invalidate following sentence of our documentation: https://docs.typo3.org/m/typo3/reference-coreapi/main/en-us/Configuration/Typo3ConfVars/DB.html > One scenario for using a separate database connection is to query data directly from a third-party application in a custom extension. Another use case is database-based caches.
  11. Benni Mack marked it work in progress 07/23/25, 14:11