Review main Merged

95325[BUGFIX] Ignore diacritics when filtering select fields

Georg Ringer CR +2 V +2 Hard +605 −11 updated 09/01/26, 19:29

Commit message

The client side filters of a "selectMultipleSideBySide" and of a
"selectTree" field compared the typed text against the labels as
written. Typing "creme" therefore did not find an option labelled
"Crème brûlée", and a category tree did not narrow to a category whose
title carries an accent, which is exactly what an editor searching a
long option list or a large tree would expect to happen.

Both sides of the comparison are now normalized first. The value is
lowercased, NFD splits an accented character into its base character
plus a combining mark, and the marks a script writes optionally are
removed: the combining diacritical marks of Latin, Greek and Cyrillic,
the Hebrew points and the Arabic harakat.

The marks Thai, Devanagari, Khmer, Lao and Japanese write obligatorily
are kept. There a mark separates one word from another rather than
accenting it, so removing every nonspacing mark would fold the Japanese
for "school" and for "cuckoo" into one value and leave a Thai filter
narrowing nothing.

Lowercasing is locale independent. A locale aware mapping follows the
browser UI locale, which is unrelated to the backend user language and
to the language of the labels, and would keep a Turkish browser from
finding "Import" when "import" is typed. Intl.Collator is unusable for
that reason as well, and because it compares whole strings rather than
searching inside one.

The Latin letters Unicode does not decompose are spelled out, so that an
alphabet folds as a whole rather than by half: otherwise Danish "å"
folds to "a" while "æ" and "ø" stay as they are. "ß" is compared as
"ss", "æ" as "ae" and "ø" as "o". The spellings that are a matter of
language, such as German "ü" to "ue", are left alone, because the
decomposition to "u" already settles those.

Characters a reader cannot see no longer keep a label from matching: a
soft hyphen and the zero width characters are dropped, and a no-break
space is compared as a plain space. Both reach labels by pasting.

This lives in the new internal module
@typo3/core/utility/text-normalizer, so that both filters and the ones
that follow normalize their input the same way.

The side by side filter folds the option labels once and caches them
beside the option list. The tree folds the node names on every
keystroke instead, because writing to a node triggers a re-render and
the nodes are replaced whenever a part of the tree is loaded.

The side by side comparison moves from RegExp to String.includes() as
well. The filter text was escaped to a literal before being handed to
the RegExp constructor anyway, so the escaping introduced with #107565
is no longer needed. The tree has compared with String.includes() since
#110482.

Review

  • Code review CR +2 Done
    • Georg Ringer +2
    • Stefan Bürk +1
    • Benjamin Kott +1
    • Benni Mack +1
  • Verified V +2 Done
    • Georg Ringer +2
    • Stefan Bürk +1
    • Benjamin Kott +1
    • Benni Mack +1
    • core-ci +1 CI

Issues 1

Files 10

Review log 1620

  1. Georg Ringer uploaded patch set 1 08/13/26, 19:17

  2. Georg Ringer commented on patch set 1 08/13/26, 19:18

    This change is ready for review.
  3. core-ci voted on patch set 1 V +1 08/13/26, 19:24 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106675

  4. Markus Klein updated the commit message in patch set 2 V +1 08/14/26, 22:23

  5. core-ci voted on patch set 2 V +1 08/14/26, 22:31 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/106733

  6. Stefan Bürk commented on patch set 2 CR -1 08/15/26, 18:07

    Thanks for finally taking #83788 on, and thanks in particular for the unit test - testing both directions (creme -> Creme brulee and crem -> Cremeschnitte) is exactly the pair that proves needle and haystack are both folded. I also verified there is no match highlighting in this component, so the usual "NFD shifted my highlight offsets" trap does not apply here. Two things hold me back, though. 1. toLocaleLowerCase() - please make it toLowerCase(). Without a locale argument this uses the browser UI locale, which has nothing to do with the backend user language or the language of the labels. On a tr-TR browser, 'I' lowercases to the dotless variant, so an option labelled Import is no longer found by typing import, and Istanbul with the dotted capital I is no longer found by typing istanbul. The old new RegExp(..., 'i') did not have that problem: ECMA-262 Canonicalize for a non-u regex is explicitly locale-independent. So this is a regression, and it is invisible to CI because the test runner's Chrome is not Turkish. toLowerCase() is also what every other filter in Build/Sources/TypeScript/ uses - backend/settings/editor.ts:454, backend/utility/collapse-state-search.ts:90, install/module/upgrade/upgrade-docs.ts:228, form/backend/form-editor-tree.ts:223 - there is currently no other toLocaleLowerCase() in the tree. 2. [BUGFIX] plus Releases: main, 14.3. #83788 is tracked as a Feature, and it was deliberately re-scoped from bugfix to feature by Oliver Hader in the issue itself back in 2018 ("I've modified [...] the scope to be a 'feature' instead of a 'bug fix'"). 14.3 is 14.3.7-dev, i.e. the v14 LTS patch line - this would ship a changed backend search semantic in a patch release, with no changelog: an editor who types Muller today to isolate it from the umlaut spelling gets both rows after the update. And if we do call it a bugfix, then 13.4 (13.4.35-dev, still actively maintained, affected identically) is missing from Releases:. My preference is [FEATURE], Releases: main, plus a Feature-83788-*.rst that also documents the sharp-s / slashed-o / ae / l-stroke limitation you already describe so honestly in the commit message. Three smaller points: * \p{Mn} strips all nonspacing marks, not just diacritics. Thai and Devanagari vowel signs are stripped too, so for those scripts the filter stops narrowing anything (two visually distinct Thai syllables fold to the same base consonant). The explicit combining-diacritical-marks range U+0300-U+036F is the narrower idiom and still covers Latin, Greek, Cyrillic and all six Vietnamese marks. * availableOptions is already cached; please cache the normalized labels next to it instead of running three string passes per option per keystroke. A selectMultipleSideBySide over sys_category easily has a few thousand options. * Consistency with your 95326: that one moves the selectTree filter to includes() but without folding, so after both changes a selectTree and a selectMultipleSideBySide in the same form answer "creme" differently. Since there is no folding helper in the TS sources yet, this might be the moment to put one in @typo3/core/ and use it from both - or at least say in the commit message that the scope is deliberately this one filter. Two questions, no action needed: was Intl.Collator({sensitivity:'base'}) considered (I agree it is unusable, no substring search - worth a sentence for the next reader), and is the Hangul side effect intentional? NFD decomposes precomposed Hangul syllables to jamo, so a syllable now matches inside a different syllable sharing the same leading jamo. Test-wise, if you touch it again: please assert the documented non-goals (strasse must not find the sharp-s spelling, mueller must not find the umlaut spelling) and add one case for the .t3js-formengine-multiselect-filter-dropdown (change) path and for <optgroup> hiding.
  7. Benjamin Kott uploaded patch set 3 CR -1 V +1 08/28/26, 10:42

  8. core-ci voted on patch set 3 V +1 08/28/26, 10:49 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107328

  9. Benjamin Kott commented on patch set 3 08/28/26, 11:00

    Reply

    done
  10. Benjamin Kott voted on patch set 3 CR +1 V +1 08/28/26, 12:54

  11. Benni Mack commented on patch set 3 CR -1 08/31/26, 09:25

    Why do we have a RST for a bugfix?
  12. Benjamin Kott commented on patch set 3 08/31/26, 13:42

    Reply

    dont mind, we can drop it, but it is also changed / improved behavior
  13. Benni Mack commented on patch set 3 CR 0 08/31/26, 13:44

    Reply

    ideally every bugfix is a changed & improved behavior. Having a new utility (text-normalizer) would IMHO justify the RST though.
  14. Georg Ringer uploaded patch set 4 CR +1 V +1 09/01/26, 08:22

  15. core-ci voted on patch set 4 V +1 09/01/26, 08:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107471

  16. Benjamin Kott voted on patch set 4 CR +1 V +1 09/01/26, 16:18

  17. Benni Mack voted on patch set 4 CR +1 V +1 09/01/26, 17:04

  18. Georg Ringer voted on patch set 4 CR +2 V +2 09/01/26, 19:15

  19. Stefan Bürk voted on patch set 4 CR +1 V +1 09/01/26, 19:16

  20. Georg Ringer merged patch set 5 09/01/26, 19:29