Review main Merged Breaking

96535[!!!][TASK] Remove UTF8filesystem and systemLocale settings

Benni Mack CR +2 V +2 Hard +405 −453 updated 10/09/26, 10:34

Commit message

Both settings existed to make PHP string functions handle UTF-8 file
names: TYPO3 switched LC_CTYPE to the configured system locale around
basename(), dirname(), pathinfo() and escapeshellarg().

The PHP manual still calls basename() and dirname() locale aware, but
the PHP source shows this does not affect UTF-8 paths:

* dirname() is zend_dirname(), which searches the separator byte by
  byte and never evaluates the locale.
* basename() is php_basename(). Since PHP 8.1 it also scans byte by
  byte when CG(ascii_compatible_locale) is set, which is the case for
  every single-byte locale (C, POSIX, ISO-8859-*) and for UTF-8. Only
  other multibyte encodings (Shift-JIS, Big5, GBK, ...) take the
  mblen() path, because their trail bytes can be 0x5C, the Windows
  directory separator. UTF-8 multibyte sequences only contain bytes
  >= 0x80 and never match a separator.
* pathinfo() is built on these two functions.

A probe with PHP 8.2, 8.4 and 8.5 on glibc confirms identical results
for UTF-8 paths in the C, C.UTF-8, en_US.UTF-8 and de_DE.UTF-8
locales. The PathUtility wrappers are therefore plain aliases now and
are deprecated in favour of the native functions.

escapeshellarg() is the one function that really depends on the
locale: php_escape_shell_arg() calls mblen() for every byte and skips
bytes that are invalid in the current locale. In the ASCII "C" locale
of glibc, every byte >= 0x80 is invalid, so "äöü.txt" turns into
".txt". The locale switch only prevented this when both settings were
configured, and indexed_search called escapeshellarg() directly.
Alpine (musl) treats every locale as UTF-8, which hides the bug.

CommandUtility::escapeShellArgument() now quotes byte-safely on POSIX
systems - single quotes, with ' escaped as '\'' - and rejects null
bytes. All locale switching is removed, file names always keep their
Unicode characters, and both settings are removed together with their
install tool checks.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Georg Ringer +1
    • Wouter Wolters +1
  • Verified V +2 Done
    • Benni Mack +2
    • Georg Ringer +1
    • Wouter Wolters +1
    • core-ci +1 CI

Issues 1

Files 53

Show 33 more files

Review log 913

  1. Benni Mack uploaded patch set 1 10/08/26, 13:45

  2. core-ci voted on patch set 1 V -1 10/08/26, 13:55 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110511

  3. Benni Mack rebased onto patch set 2 V -1 10/08/26, 14:57

  4. core-ci voted on patch set 2 V +1 10/08/26, 15:06 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110513

  5. Benni Mack uploaded patch set 3 V +1 10/08/26, 15:11

  6. Benni Mack marked it ready for review 10/08/26, 15:13

  7. core-ci voted on patch set 3 V -1 10/08/26, 15:23 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110514

  8. Benni Mack uploaded patch set 4 V -1 10/08/26, 17:16

  9. core-ci voted on patch set 4 V +1 10/08/26, 17:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110518

  10. Wouter Wolters voted on patch set 4 CR +1 V +1 10/08/26, 18:05

  11. Georg Ringer voted on patch set 4 CR +1 V +1 10/08/26, 18:09

  12. Benni Mack voted on patch set 4 CR +2 V +2 10/09/26, 10:34

  13. Benni Mack merged patch set 5 10/09/26, 10:34