#108430TCA type "json" does not preserve Unicode characters - encodes as \uXXXX escape sequences
Description
Description
When storing JSON data via DataHandler, Unicode characters are converted to \uXXXX escape sequences. This happens because the internal re-encoding does not use the JSON_UNESCAPED_UNICODE flag.
While both representations are technically valid JSON, this causes several issues:
Data comparison fails: When updating a record with the same logical data but different encoding (e.g., “Meßgaspumpe” vs “Me\u00dfgaspumpe”), DataHandler compares the decoded arrays and sees no change, silently skipping the update
Increased storage size: Escaped sequences use 6 bytes per character vs 1-4 bytes for UTF-8
Reduced readability: Database content is harder to read/debug
Inconsistent behavior: Input encoding is not preserved
Steps to Reproduce
Create a TCA field with ‘type’ => ‘json’
Save a record via DataHandler with JSON containing Unicode characters:
$dataHandler->start([
'my_table' => [
'NEW1' => [
'json_field' => '{"name": "Meßgaspumpe", "city": "München"}'
]
]
], []);
$dataHandler->process_datamap();
Check the stored value in the database
Expected Result
json{"name": "Meßgaspumpe", "city": "München"}
Actual Result
json{"name": "Me\u00dfgaspumpe", "city": "M\u00fcnchen"}
Code Reference
The issue originates in TYPO3\CMS\Core\DataHandling\DataHandler::checkValueForJson() which decodes the JSON string into a PHP array. The subsequent re-encoding to JSON (in the database layer) uses json_encode() without the JSON_UNESCAPED_UNICODE flag.
Suggested Solution
Either:
A) Preserve original string encoding – If input is already valid JSON, pass it through without decode/encode cycle
B) Use proper encoding flags – When re-encoding, use:
json_encode($value, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES)
C) Add TCA configuration – Allow developers to specify encoding flags:
'json_field' => [
'config' => [
'type' => 'json',
'jsonEncodeOptions' => JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES,
],
]
Option B would be the simplest fix with the least risk, as JSON_UNESCAPED_UNICODE is generally the preferred default for UTF-8 databases.
Impact
This affects any extension storing multilingual or special character content in JSON fields, particularly relevant for international TYPO3 installations.
Reviews 2
History 312
-
Christopher Kussin commented <span data-processor="relativetime" data-value="2025-12-09T04:27:01+00:00" > 12/09/25, 04:27 </span>
-
Stefan Bürk changed assigned to Stefan Bürk <span data-processor="relativetime" data-value="2025-12-10T05:50:38+00:00" > 12/10/25, 05:50 </span>
-
Gerrit Code Review pushed patch set 1 for main to change 92719 status
NewUnder Review <span data-processor="relativetime" data-value="2026-02-06T14:39:06+00:00" > 02/06/26, 14:39 </span> -
Gerrit Code Review pushed patch set 2 for main to change 92719 <span data-processor="relativetime" data-value="2026-02-06T15:46:47+00:00" > 02/06/26, 15:46 </span>
-
Gerrit Code Review pushed patch set 3 for main to change 92719 <span data-processor="relativetime" data-value="2026-02-06T18:29:02+00:00" > 02/06/26, 18:29 </span>
-
Gerrit Code Review pushed patch set 4 for main to change 92719 <span data-processor="relativetime" data-value="2026-04-22T06:40:10+00:00" > 04/22/26, 06:40 </span>
-
Gerrit Code Review pushed patch set 5 for main to change 92719 <span data-processor="relativetime" data-value="2026-07-22T09:18:02+00:00" > 07/22/26, 09:18 </span>
-
Gerrit Code Review pushed patch set 6 for main to change 92719 <span data-processor="relativetime" data-value="2026-08-27T17:26:05+00:00" > 08/27/26, 17:26 </span>
-
Gerrit Code Review pushed patch set 7 for main to change 92719 <span data-processor="relativetime" data-value="2026-09-01T19:21:41+00:00" > 09/01/26, 19:21 </span>
-
Gerrit Code Review pushed patch set 1 for 14.3 to change 95529 <span data-processor="relativetime" data-value="2026-09-01T19:43:25+00:00" > 09/01/26, 19:43 </span>
-
Gerrit Code Review pushed patch set 2 for 14.3 to change 95529 <span data-processor="relativetime" data-value="2026-09-01T20:04:12+00:00" > 09/01/26, 20:04 </span>
-
Georg Ringer commented and changed status
Under ReviewResolved done0 %100 % <span data-processor="relativetime" data-value="2026-09-01T20:20:08+00:00" > 09/01/26, 20:20 </span>Applied in changeset commit:b5d0083ab9dce17299984f853d6555e0bca6b38f.
“Data comparison fails: When updating a record with the same logical data but different encoding (e.g., “Meßgaspumpe” vs “Me\u00dfgaspumpe”), DataHandler compares the decoded arrays and sees no change, silently skipping the update”
The DataHandler mentioned here is not the Typo3 DataHandler. Our own system also has a DataHandler so lets call it “CustomDataHandler”