Review main Merged

73636[BUGFIX] Use proper Postgres types on inserting / updating rows

Stefan Bürk CR +2 V +2 Normal +72 −12 updated 02/22/22, 08:33

Commit message

In early stage of TYPO3v11 the storage format for data
in the TCA ctrl transOrigDiffSourceField database field,
often called l18n_diffsource changed from PHP serialized
data to JSON encoded storage format with #91906, mainly
to mitigate general security risks.

doctrine/dbal enquotes json data provided as string with
backslashes if not the correct field type is provided,
which emits a postgres sql error exception when inserting
or updating these fields, which are of type 'bytea' when
dbms Postgres is used.

Various parts of the core deal with arbitrary tables and
don't know if a column is int, text or lob, or whatever.
Those are blindly updated/inserted, resulting in Postgres
saying "no".

MSSQL has been named pickier than postgres in the past on
that for similar issues with these fields. To solve this
for MSSQL a expensive workaround on several places through
the core with #81498 to get a working state and leaving it
open to find a better way with a eventual cache-layer for
these database field schema informations, thus unsolved yet.

As core suffers now from the same issue for Postgres since
the changed storage format, this patch adopts the choosen
MSSQL solution for Postgres. This is a first bugfix solution
with the tradeoff to be more expensive but working.

Additionally one test case is added to cover this issue
basicly but should be extended and further tightend in a
dedicated patch.

Improving the schema handling should be picked up and proper
engineered in a another dedicated patch.

Review

  • Code review CR +2 Done
    • Oli Bartsch +2
    • Benni Mack +1
  • Verified V +2 Done
    • Oli Bartsch +2
    • Benni Mack +1
    • core-ci +1 CI

Issues 3

Files 7

Review log 1014

  1. Stefan Bürk uploaded patch set 1 02/21/22, 21:47

  2. Stefan Bürk updated the commit message in patch set 2 02/21/22, 21:57

  3. core-ci voted on patch set 1 V +1 02/21/22, 21:59 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/24015

  4. Stefan Bürk updated the commit message in patch set 3 02/21/22, 22:03

  5. core-ci voted on patch set 2 V +1 02/21/22, 22:07 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/24016

  6. core-ci voted on patch set 3 V +1 02/21/22, 22:11 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/24017

  7. Benni Mack voted on patch set 3 V +1 CR +1 02/21/22, 22:46

  8. Stefan Bürk uploaded patch set 4 02/22/22, 02:33

  9. core-ci voted on patch set 4 V +1 02/22/22, 02:41 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/24021

  10. Stefan Bürk commented on patch set 4 02/22/22, 02:47

    Added one testcase as requests in slack by Oliver Bartsch (thanks for nailing me on this). This testcase only covers the change in DataHandler for now. If the change is reverted the test failes with the message that the expected dataset do not match. This is basicly correct, but we do not get the "error message" directly which is displayed in the backend for now. DataHandler::errorLog[] is not checked and converted to test failures / exception or something similar, thus the database exception / error log entry got swallowed in TF ActionService methods(). Will discuss this and some other findings with lolli the next days, as we have some points to tightend the whole testing even more (thanks Oliver, would not have found these without your test request). Leaving this one testcase which basicly covers this with a error message if broken again, even if missleading for now.
  11. Benni Mack voted on patch set 4 V +1 CR +1 02/22/22, 08:01

  12. Oli Bartsch commented on patch set 4 V +2 CR +2 02/22/22, 08:32

    Could we somehow test this?
  13. Oli Bartsch merged patch set 5 02/22/22, 08:32

  14. Oli Bartsch commented on patch set 5 02/22/22, 08:33

    Reply

    Done.