#109828SlugNormalizer::normalize() silently drops non-Latin Unicode characters (e.g. CJK/Chinese)
Description
When generating a slug from a string that contains non-Latin Unicode characters (e.g. Chinese, Japanese, Korean), those characters are silently dropped and the resulting slug is incomplete.
Example:
Input: PFAS Free 防水透气膜
Expected slug: pfas-free-防水透气膜
Actual slug: pfas-free
Root cause:
In TYPO3\CMS\Core\Slug\SlugNormalizer::normalize(), the call to $this->charsetConverter->specCharsToASCII(‘utf-8’, $value) converts special characters to their ASCII equivalents. This works well for European characters (e.g.
ä → ae) but has no mapping for CJK or other non-Latin scripts, so those characters are silently replaced with an empty string.
The subsequent regex preg_replace(‘/[^\p{L}\p{M}0-9\/…]/u’, ‘’, $value) uses the \p{L} Unicode letter class which would preserve CJK characters — but by that point they have already been stripped by specCharsToASCII.
Steps to reproduce:
$normalizer = GeneralUtility::makeInstance(\TYPO3\CMS\Core\Slug\SlugNormalizer::class);
$result = $normalizer->normalize(‘PFAS Free 防水透气膜’);
// Returns: “pfas-free” — expected: “pfas-free-防水透气膜”
Suggested fix:
Only call specCharsToASCII() on the ASCII-range portion of the string, or skip it entirely for code points outside the Latin script range. The \p{L} regex that follows is already sufficient to strip truly invalid characters
while preserving valid Unicode letters.
TYPO3 version: 13.xReviews 0
No change resolves or names this issue in its commit message, and no comment links one.