Issue Bug Closed

#93680wrong handling of utf8 of fal filenames when importing youtube videos

Peter Linzenkirchner 03/08/21, 16:40 updated 04/13/24, 22:49

Description

TYPO3 9.5.24
UTF8 Filesystem in Installtool active
de_DE-UTF-8 systemLocale in Installtool
Server Max OS X

The filename of the video contains an emoji:

Module filelist, add external url results in this response:
Added media
20_Unsere_Erfolgsstory__{emoticon – redmine cannot save it}.youtube

The file is on the disk with exactly this name – Mac OS X is able to manage this characters. But TYPO3 shows the wrong filenames in the file module:

20_Unsere_Erfolgsstory__ – the rest is missing.

So the systems and TYPO3 manages the download and the saving of the file correctly but the identifier in sys_file is wrong.

Reviews 0

No change resolves or names this issue in its commit message, and no comment links one.

History 7

  1. Guido Schmechel commented 03/13/21, 22:48

    Test in v11:

    - File is saved correctly, emoji replaced by “_“
    - typo3/sysext/core/Classes/Resource/OnlineMedia/Helpers/AbstractOEmbedHelper.php throws an exception at the insert with “Incorrect string value”.

  2. Martin Kutschker commented 05/07/21, 16:11

    Is the DB set to use utf8_mb4?

    With the older mb3 none of the emojis can be saved. The reason is that mb3 does NOT cover the complete set of Unicode codepoints.

  3. Riccardo De Contardi changed status New Needs Feedback 08/08/21, 08:07

  4. Peter Linzenkirchner commented 08/08/21, 10:36

    The database is (and was) utf8mb4_general_ci but not all table or field defintions were utf8mb4_general_ci. In a test installation i changed radically all table and field collations to utf8mb4_general_ci and tried again, but nothing changes.
    • the video can be loaded but not played
    • it is impossible to delete the broken video file in the file module of typo3

    You can check it with this video: https://www.youtube.com/watch?v=FsJiVC2nBX8

  5. Riccardo De Contardi commented 09/17/22, 08:45

    I tried to reproduce the problem on the following:

    - latest TYPO3 master 12.0.0-dev
    - TYPO3 11.5.16
    - TYPO3 10.4.32

    Under the following conditions:

    - I use TYPO3 on MACOS with MAMP
    - The database collation is utf8mb4_general_ci

    Test procedure

    I created a new “text and media” content element and used “add media by URL”; The result is that *the video is correctly shown on frontend*
    Moreover, editing the “Video Metadata” > “Title”, the field shows the emoji

    I checked Filelist > user_upload

    - on the file list the video has been saved as 20_Jahre_ITC_GmbH_-_Unsere_Erfolgsstory______.youtube
    - if I edit it, the title filed shows the emoji (*)

    Then I
    - removed the video from the content element (remove the sys_file_reference)
    - Filelist > user_upload I tried to delete the video “file” and it has been succesfully deleted

    (*) (this actually happened on the latest TYPO3 master because EXT:filemetadata was installed)

    Is this sufficient or a different test is necessary?

  6. Stefan Froemken commented 12/10/22, 13:03

    This is not a TYPO3 problem, this is a configuration problem.
    First of all you have to set all DB tables and columns to utf8mb4_unicode_ci. That’s needed to allow storing values like icons and smileys.
    Second: You have to set DB charset in your LocalConfiguration to: utf8mb4. That’s needed to transfer the real MB4 value (smiley) to the database. In many TYPO3 installations this value is still “utf8” which removes the 4th byte for smileys while query.
    Third: If possible I prefer enabling SYS UTF8filesystem to 1

    With that configuration the problem is gone.

    IMO that issue can be closed now.

  7. Riccardo De Contardi commented and changed status Needs Feedback Closed 04/13/24, 22:49

    There has been no feedback since a long time; plus, considering what Stefan Froemken (thank you) wrote, I assume that it is safe to close this issue for now.

    If you think that this is the wrong decision or there is still work to be done please open a new issue with a reference to this one.

    Sorry for the very late reply.

    Thank you.