Issue Bug Closed

#109828SlugNormalizer::normalize() silently drops non-Latin Unicode characters (e.g. CJK/Chinese)

Torsten Niederberger 05/13/26, 11:56 updated 05/13/26, 15:08

Description

When generating a slug from a string that contains non-Latin Unicode characters (e.g. Chinese, Japanese, Korean), those characters are silently dropped and the resulting slug is incomplete.

Example:
Input: PFAS Free 防水透气膜
Expected slug: pfas-free-防水透气膜
Actual slug: pfas-free
Root cause:
In TYPO3\CMS\Core\Slug\SlugNormalizer::normalize(), the call to $this->charsetConverter->specCharsToASCII(‘utf-8’, $value) converts special characters to their ASCII equivalents. This works well for European characters (e.g.
ä → ae) but has no mapping for CJK or other non-Latin scripts, so those characters are silently replaced with an empty string.
The subsequent regex preg_replace(‘/[^\p{L}\p{M}0-9\/…]/u’, ‘’, $value) uses the \p{L} Unicode letter class which would preserve CJK characters — but by that point they have already been stripped by specCharsToASCII.
Steps to reproduce:
$normalizer = GeneralUtility::makeInstance(\TYPO3\CMS\Core\Slug\SlugNormalizer::class);
$result = $normalizer->normalize(‘PFAS Free 防水透气膜’);
// Returns: “pfas-free” — expected: “pfas-free-防水透气膜”
Suggested fix:
Only call specCharsToASCII() on the ASCII-range portion of the string, or skip it entirely for code points outside the Latin script range. The \p{L} regex that follows is already sufficient to strip truly invalid characters
while preserving valid Unicode letters.
TYPO3 version: 13.x

Reviews 0

No change resolves or names this issue in its commit message, and no comment links one.

History 2

  1. Garvin Hicking commented and changed status New Closed <span data-processor="relativetime" data-value="2026-05-13T15:08:19+00:00" > 05/13/26, 15:08 </span>

    This is a duplicate of #20612 and was alao subject of a budhet proposal:

    https://talk.typo3.org/t/not-only-support-german-transliteration-in-typo3-core/5056

    Sadly the fix is not as straight ahead as you can see from the attached patch to that issue. If you’d like to refresh the patch and work on it that would of course be much appreciated.