Review 14.3 Merged

95524[BUGFIX] Ignore diacritics when filtering select fields

Georg Ringer CR +2 V +2 Hard +605 −11 updated 09/01/26, 19:29

Commit message

The client side filters of a "selectMultipleSideBySide" and of a
"selectTree" field compared the typed text against the labels as
written. Typing "creme" therefore did not find an option labelled
"Crème brûlée", and a category tree did not narrow to a category whose
title carries an accent, which is exactly what an editor searching a
long option list or a large tree would expect to happen.

Both sides of the comparison are now normalized first. The value is
lowercased, NFD splits an accented character into its base character
plus a combining mark, and the marks a script writes optionally are
removed: the combining diacritical marks of Latin, Greek and Cyrillic,
the Hebrew points and the Arabic harakat.

The marks Thai, Devanagari, Khmer, Lao and Japanese write obligatorily
are kept. There a mark separates one word from another rather than
accenting it, so removing every nonspacing mark would fold the Japanese
for "school" and for "cuckoo" into one value and leave a Thai filter
narrowing nothing.

Lowercasing is locale independent. A locale aware mapping follows the
browser UI locale, which is unrelated to the backend user language and
to the language of the labels, and would keep a Turkish browser from
finding "Import" when "import" is typed. Intl.Collator is unusable for
that reason as well, and because it compares whole strings rather than
searching inside one.

The Latin letters Unicode does not decompose are spelled out, so that an
alphabet folds as a whole rather than by half: otherwise Danish "å"
folds to "a" while "æ" and "ø" stay as they are. "ß" is compared as
"ss", "æ" as "ae" and "ø" as "o". The spellings that are a matter of
language, such as German "ü" to "ue", are left alone, because the
decomposition to "u" already settles those.

Characters a reader cannot see no longer keep a label from matching: a
soft hyphen and the zero width characters are dropped, and a no-break
space is compared as a plain space. Both reach labels by pasting.

This lives in the new internal module
@typo3/core/utility/text-normalizer, so that both filters and the ones
that follow normalize their input the same way.

The side by side filter folds the option labels once and caches them
beside the option list. The tree folds the node names on every
keystroke instead, because writing to a node triggers a re-render and
the nodes are replaced whenever a part of the tree is loaded.

The side by side comparison moves from RegExp to String.includes() as
well. The filter text was escaped to a literal before being handed to
the RegExp constructor anyway, so the escaping introduced with #107565
is no longer needed. The tree has compared with String.includes() since
#110482.

Review

  • Code review CR +2 Done
    • Georg Ringer +2
  • Verified V +2 Done
    • Georg Ringer +2
    • core-ci +1 CI

Issues 1

Files 10

Review log 34

  1. Georg Ringer commented on patch set 1 09/01/26, 19:15

    Cherry Picked from branch main.
  2. core-ci voted on patch set 1 V +1 09/01/26, 19:27 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/107507

  3. Georg Ringer voted on patch set 1 CR +2 V +2 09/01/26, 19:29

  4. Georg Ringer merged patch set 2 09/01/26, 19:29