Review main Merged

85658[TASK] Mitigate deprecated Doctrine DBAL connection options

Stefan Bürk CR +2 V +2 Normal +129 −37 updated 10/14/24, 14:08

Commit message

Passing `tableoptions` through the connection params
introduced with #80398 violates the PHPStan annotation
since quite some time. To mitigate this, automatic
transformation along with an @todo has been added to
`ConnectionPool` with #97961.

Doctrine DBAL meanwhile switched from `collate` to
`collation` for internal table and column collate
settings since `doctrine/dbal:3.3.2` [1] putting a
backwards compatible layer into place. This has been
removed with v4.

Setting default table options could have been set
manually or are set as part if the install process,
but are not guaranteed in all cases.

Default connection charset is ensured in several
places: In ConnectionPool and during installation
process. In some cases respecting MySQL/MariaDB
differences, but not in other places.

Deprecated connection parameter options needs to be
avoided and migrated and charset should be ensured
the same way in all places.

This change:

* Modifies `ConnectionPool` to migrate `tableoptions`
  to `defaultTableOptions` with `tableoptions` taking
  higher priority for now. MariaDB/MySQL only.
* Modifies `ConnectionPool` to migrate `collate` table
  option to `collation` with `collate` having higher
  priority for now. MariaDB/MySQL only.
* Modifies `ConnectionPool` to remove `collate` option.
* Modifies `ConnectionPool` to remove `tableoptions`
  and `defaultTableOptions` for connections not made
  towards MySQL/MariaDB.
* Modifes `ConnectionPool` to use `utf8mb4` charset for
  MySQL/MariaDB connections and `utf8` for SQLite and
  PostgreSQL.
* Modifies EXT:install `SetupDatabaseService` to set
  `defaultTableOptions` charset and collation along
  with connection `charset` settings during the setup
  process.
* Modifes `SetupDatabaseService` to use `utf8mb4` charset
  for MySQL/MariaDB connections and `utf8` for SQLite and
  PostgreSQL.

Old options will be deprecated in a dedicated change.
Further, `typo3/testing-framework` has been prepared as
well with [2][3] and #105288.

[1] https://github.com/doctrine/dbal/pull/5246
[2] https://github.com/TYPO3/testing-framework/pull/629
[3] https://github.com/TYPO3/testing-framework/pull/628

Review

  • Code review CR +2 Done
    • Stefan Bürk +2
    • Garvin Hicking +1
    • Anja Leichsenring +1
    • Christian Kuhn +1
  • Verified V +2 Done
    • Stefan Bürk +2
    • Garvin Hicking +1
    • Anja Leichsenring +1
    • Christian Kuhn +1
    • core-ci +1 CI

Issues 4

Files 2

Review log 3148

Show 1323 earlier messages
  1. Stefan Bürk uploaded patch set 1 08/16/24, 13:51

  2. core-ci voted on patch set 1 V -1 08/16/24, 14:02 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63584

  3. core-ci voted on patch set 1 V +1 08/16/24, 15:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63584

  4. Stefan Bürk uploaded patch set 2 V +1 08/17/24, 12:31

  5. core-ci voted on patch set 2 V -1 08/17/24, 12:43 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63653

  6. core-ci voted on patch set 2 V +1 08/17/24, 12:55 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/63653

  7. Andreas Kienast commented on patch set 2 08/19/24, 10:04

    Can't we achieve the same with a configuration re-write in SilentConfigurationUpgradeService.php?
  8. Stefan Bürk commented on patch set 2 08/26/24, 13:11

    Reply

    Just thought about that too. However, that would not tackle dynamic configuration set for example in additional.php - only that in the settings.php. Still, I guess I add a SilentConfigurationUpgradeService but would still keep the transformation stuff (and later deprecation). What do you think ?
  9. Stefan Bürk commented on patch set 2 08/26/24, 16:08

    Reply

    Decided to add the silentconfiguration change thingy to the deprecation patch (main/v13 only). The analyzer workaround will be removed in a follow-up change anyway than (this is just one in a series of patches v12/v13 and v13 only). Anyway, thanks for also mentioning the silent upgrade thingy.
  10. Stefan Bürk uploaded patch set 3 V +1 10/12/24, 16:11

  11. core-ci voted on patch set 3 V -1 10/12/24, 16:18 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67652

  12. Stefan Bürk rebased onto patch set 4 V -1 10/12/24, 16:26

  13. core-ci voted on patch set 4 V +1 10/12/24, 16:33 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67654

  14. Stefan Bürk uploaded patch set 5 V +1 10/12/24, 18:13

  15. core-ci voted on patch set 5 V +1 10/12/24, 18:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67661

  16. Stefan Bürk updated the commit message in patch set 6 V +1 10/12/24, 18:24

  17. core-ci voted on patch set 6 V +1 10/12/24, 18:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67662

  18. Stefan Bürk commented on patch set 6 CR -1 10/12/24, 18:56

    on it ...
  19. Stefan Bürk uploaded patch set 7 CR -1 V +1 10/12/24, 19:38

  20. core-ci voted on patch set 7 V +1 10/12/24, 19:43 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67665

  21. Garvin Hicking commented on patch set 7 10/12/24, 21:36

    ConnectionPool.php:231

    Is that dynamic unicode_ci suffix really safe to use? If one would use latin1 as charset (not that one still should), would result in latin1_unicode_ci which doesn't exist. I also think it's "utf8" not "utf-8" for mysql, but I haven't checked if dbal does some normalization there? And for that the collation would be "utf8_general_ci" in mysql. Maybe better use a fixed assignment (switch/match)? Please just resolve my comment, if dbal 4.2 does magic here I wasn't aware of :-)
  22. Stefan Bürk uploaded patch set 8 V +1 10/13/24, 09:47

  23. core-ci voted on patch set 8 V +1 10/13/24, 09:52 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67681

  1. Stefan Bürk uploaded patch set 9 V +1 10/13/24, 10:47

  2. core-ci voted on patch set 9 V -1 10/13/24, 10:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67685

  3. Stefan Bürk uploaded patch set 10 V -1 10/13/24, 11:31

  4. core-ci voted on patch set 10 V -1 10/13/24, 11:39 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67687

  5. Stefan Bürk uploaded patch set 11 V -1 10/13/24, 12:35

  6. core-ci voted on patch set 11 V +1 10/13/24, 12:44 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67692

  7. Stefan Bürk commented on patch set 11 10/13/24, 13:07

    ConnectionPool.php:231 Reply

    Really good point, thanks for pointing to it. This change is about migrationg old options the new, and should not deal with changing already settings behaviour at all. Thus, I removed the handling here in ConnectionPool again for defaultTableOptions, and just left to set the normal connection charset either `utf8`(sqlite/postgres) or `utf8mb4`(MySQL/MariaDB) in case none is set, mainly like. How to deal with completly missing defaultTableOptions should be handled in a dedicated change, which may also be a change in behaviour anyway and was not ment to be introduced with this change at all. Beside that, I changed the `SetupDatabaseService` again to purly set only default values when not present and not mixing/dealing with subsets or unsafe combination building for now. Doctrine DBAL does not much magic here at all, even more dropped applying stuff to columns all together. TYPO3 does some things in that area for the database analyzer, which has the one or other bug I tend to solve as one of the next steps but wanted to streamline the migration towards the new options instead of dealing with old/new options matrix variants in the analyzer. There, only the new version should be available. Thus, resolving this comment now.
  8. Stefan Bürk uploaded patch set 12 V +1 10/13/24, 13:12

  9. Stefan Bürk rebased onto patch set 13 10/13/24, 13:19

  10. core-ci voted on patch set 12 V +1 10/13/24, 13:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67696

  11. core-ci voted on patch set 13 V +1 10/13/24, 13:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67698

  12. Anja Leichsenring voted on patch set 13 CR +1 V +1 10/13/24, 13:48

  13. Garvin Hicking commented on patch set 13 10/13/24, 18:10

    ConnectionPool.php:218

    defaultTableOptions will always exist due to migrateTableOptionsToDefaultTableOptions? That would allow the next if-statement to get "else" for simplicity?

    ConnectionPool.php:249

    The method above does not have this "(string)", could this be unified? Also that driverSupportsDefaultTableOptions is slightly different in the two methods?

    ConnectionPool.php:274

    I'm unsure here, if I read this correctly, utf8 was used for non-fresh installs that didnt specify a charset, but now utf8mb4 would be used? could that break non-migrated instances then? I believe though using utf8mb4 as a connection charset should be downwards compatible to utf8, right?!

    SetupDatabaseService.php:234

    This feels like a bugfix, but wonder if there ever was intent to return env instead of adapted array?!

    SetupDatabaseService.php:715

    For non mysql instances this was "utf-8" and not "utf8" before, is tbis intentional?
  14. Stefan Bürk uploaded patch set 14 CR +1 V +1 10/14/24, 04:24

  15. Stefan Bürk commented on patch set 14 10/14/24, 04:24

    ConnectionPool.php:218 Reply

    Currently, but it is not ensured that this method may be called from another point not having it and wanted to make clear that it should remove a existing one. `unset()` is safe to call when not existing, going to simplyfy it and remove the array_key_exist check in this condition. > That would allow the next if-statement to get "else" for simplicity? No, this one should remove the key for not supported drivers at all cost, the check below only when empty for supported drivers. Used early return now here, which made the code below a little more straight forward.

    ConnectionPool.php:249 Reply

    Done

    ConnectionPool.php:274 Reply

    utf8 for postgres/sqlite, utf8mb4 for mysql/mariadb. Translates to utf8mb4 or utf8mb3 depending versions of mariadb (mysql should be safe) in default installations. webinstaller had both in it for some time and written to settings.php, excpt it has been short circut and written only as utf8 (having a additional.php using ddev for example). Can occur if settings/additional are completly hand made or connection charset removed at all, otherwise at least utf8 is already written to the settings.php file during setups. Can it occur ? Yes, but I would count that nearly neglitable. At least that should not discussed here but to the oved place L253-255. However, will remove it from above and redoing that in 14.0 as breaking change for that 0.000001 permille.

    SetupDatabaseService.php:234 Reply

    Yes and no, more no. Do you insist of making that in a dedicated change ? Will extract that and make that fix in 13.4.1. If this change does not go in I cannot fix a more important bug in the database analyser, having that here directly in improves the overall experience.

    SetupDatabaseService.php:715 Reply

    Yes, this was intentional. See commit message: > Modifes `ConnectionPool` to use `utf8mb4` charset for > MySQL/MariaDB connections and `utf8` for SQLite and > PostgreSQL. Note both mapped `utf-8` to `utf8`, but > is here now streamlined to match other places in the > TYPO3 core and the correct naming for both vendors. Will add the same phrasing mentioning the place in the SetupDatabaseSerivce.
  16. core-ci voted on patch set 14 V +1 10/14/24, 04:29 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67726

  17. Garvin Hicking commented on patch set 14 CR +1 10/14/24, 06:51

    SetupDatabaseService.php:234 Reply

    No, just wanted to understand that part properly, would be fine by me to include this
  18. Christian Kuhn voted on patch set 14 CR +1 V +1 10/14/24, 11:05

  19. Christian Kuhn uploaded patch set 15 CR +1 V +1 10/14/24, 13:55

  20. core-ci voted on patch set 15 V +1 10/14/24, 14:03 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/67759

  21. Anja Leichsenring voted on patch set 15 CR +1 V +1 10/14/24, 14:05

  22. Garvin Hicking voted on patch set 15 CR +1 V +1 10/14/24, 14:06

  23. Christian Kuhn voted on patch set 15 CR +1 V +1 10/14/24, 14:06

  24. Stefan Bürk voted on patch set 15 CR +2 V +2 10/14/24, 14:07

  25. Stefan Bürk merged patch set 16 10/14/24, 14:08