Issue Bug Resolved

#108430TCA type "json" does not preserve Unicode characters - encodes as \uXXXX escape sequences

Christopher Kussin 12/09/25, 04:22 updated 09/01/26, 20:20

Description

Description

When storing JSON data via DataHandler, Unicode characters are converted to \uXXXX escape sequences. This happens because the internal re-encoding does not use the JSON_UNESCAPED_UNICODE flag.
While both representations are technically valid JSON, this causes several issues:

Data comparison fails: When updating a record with the same logical data but different encoding (e.g., “Meßgaspumpe” vs “Me\u00dfgaspumpe”), DataHandler compares the decoded arrays and sees no change, silently skipping the update
Increased storage size: Escaped sequences use 6 bytes per character vs 1-4 bytes for UTF-8
Reduced readability: Database content is harder to read/debug
Inconsistent behavior: Input encoding is not preserved

Steps to Reproduce

Create a TCA field with ‘type’ => ‘json’
Save a record via DataHandler with JSON containing Unicode characters:

$dataHandler->start([
       'my_table' => [
           'NEW1' => [
               'json_field' => '{"name": "Meßgaspumpe", "city": "München"}'
           ]
       ]
   ], []);
   $dataHandler->process_datamap();

Check the stored value in the database

Expected Result

json{"name": "Meßgaspumpe", "city": "München"}

Actual Result

json{"name": "Me\u00dfgaspumpe", "city": "M\u00fcnchen"}

Code Reference

The issue originates in TYPO3\CMS\Core\DataHandling\DataHandler::checkValueForJson() which decodes the JSON string into a PHP array. The subsequent re-encoding to JSON (in the database layer) uses json_encode() without the JSON_UNESCAPED_UNICODE flag.

Suggested Solution

Either:
A) Preserve original string encoding – If input is already valid JSON, pass it through without decode/encode cycle
B) Use proper encoding flags – When re-encoding, use:

json_encode($value, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES)

C) Add TCA configuration – Allow developers to specify encoding flags:

'json_field' => [
    'config' => [
        'type' => 'json',
        'jsonEncodeOptions' => JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES,
    ],
]

Option B would be the simplest fix with the least risk, as JSON_UNESCAPED_UNICODE is generally the preferred default for UTF-8 databases.

Impact

This affects any extension storing multilingual or special character content in JSON fields, particularly relevant for international TYPO3 installations.

Reviews 2

History 312

  1. Christopher Kussin commented <span data-processor="relativetime" data-value="2025-12-09T04:27:01+00:00" > 12/09/25, 04:27 </span>

    “Data comparison fails: When updating a record with the same logical data but different encoding (e.g., “Meßgaspumpe” vs “Me\u00dfgaspumpe”), DataHandler compares the decoded arrays and sees no change, silently skipping the update”

    The DataHandler mentioned here is not the Typo3 DataHandler. Our own system also has a DataHandler so lets call it “CustomDataHandler”

  2. Georg Ringer commented and changed status Under Review Resolved done 0 % 100 % <span data-processor="relativetime" data-value="2026-09-01T20:20:08+00:00" > 09/01/26, 20:20 </span>

    Applied in changeset commit:b5d0083ab9dce17299984f853d6555e0bca6b38f.