#93680wrong handling of utf8 of fal filenames when importing youtube videos
Description
TYPO3 9.5.24
UTF8 Filesystem in Installtool active
de_DE-UTF-8 systemLocale in Installtool
Server Max OS X
The filename of the video contains an emoji:
Module filelist, add external url results in this response:
Added media
20_Unsere_Erfolgsstory__{emoticon – redmine cannot save it}.youtube
The file is on the disk with exactly this name – Mac OS X is able to manage this characters. But TYPO3 shows the wrong filenames in the file module:
20_Unsere_Erfolgsstory__ – the rest is missing.
So the systems and TYPO3 manages the download and the saving of the file correctly but the identifier in sys_file is wrong.
Reviews 0
No change resolves or names this issue in its commit message, and no comment links one.
History 7
-
Guido Schmechel commented 03/13/21, 22:48
-
Martin Kutschker commented 05/07/21, 16:11
Is the DB set to use utf8_mb4?
With the older mb3 none of the emojis can be saved. The reason is that mb3 does NOT cover the complete set of Unicode codepoints.
-
Riccardo De Contardi changed status
NewNeeds Feedback 08/08/21, 08:07 -
Peter Linzenkirchner commented 08/08/21, 10:36
The database is (and was) utf8mb4_general_ci but not all table or field defintions were utf8mb4_general_ci. In a test installation i changed radically all table and field collations to utf8mb4_general_ci and tried again, but nothing changes.- the video can be loaded but not played
- it is impossible to delete the broken video file in the file module of typo3
You can check it with this video: https://www.youtube.com/watch?v=FsJiVC2nBX8
-
Riccardo De Contardi commented 09/17/22, 08:45
I tried to reproduce the problem on the following:
- latest TYPO3 master 12.0.0-dev
- TYPO3 11.5.16
- TYPO3 10.4.32Under the following conditions:
- I use TYPO3 on MACOS with MAMP
- The database collation is utf8mb4_general_ciTest procedure
I created a new “text and media” content element and used “add media by URL”; The result is that *the video is correctly shown on frontend*
Moreover, editing the “Video Metadata” > “Title”, the field shows the emojiI checked Filelist > user_upload
- on the file list the video has been saved as
20_Jahre_ITC_GmbH_-_Unsere_Erfolgsstory______.youtube
- if I edit it, thetitlefiled shows the emoji (*)Then I
- removed the video from the content element (remove the sys_file_reference)
- Filelist > user_upload I tried to delete the video “file” and it has been succesfully deleted(*) (this actually happened on the latest TYPO3 master because EXT:filemetadata was installed)
Is this sufficient or a different test is necessary?
-
Stefan Froemken commented 12/10/22, 13:03
This is not a TYPO3 problem, this is a configuration problem.
First of all you have to set all DB tables and columns to utf8mb4_unicode_ci. That’s needed to allow storing values like icons and smileys.
Second: You have to set DB charset in your LocalConfiguration to: utf8mb4. That’s needed to transfer the real MB4 value (smiley) to the database. In many TYPO3 installations this value is still “utf8” which removes the 4th byte for smileys while query.
Third: If possible I prefer enabling SYS UTF8filesystem to 1With that configuration the problem is gone.
IMO that issue can be closed now.
-
Riccardo De Contardi commented and changed status
Needs FeedbackClosed 04/13/24, 22:49There has been no feedback since a long time; plus, considering what Stefan Froemken (thank you) wrote, I assume that it is safe to close this issue for now.
If you think that this is the wrong decision or there is still work to be done please open a new issue with a reference to this one.
Sorry for the very late reply.
Thank you.
Test in v11:
- File is saved correctly, emoji replaced by “_“
- typo3/sysext/core/Classes/Resource/OnlineMedia/Helpers/AbstractOEmbedHelper.php throws an exception at the insert with “Incorrect string value”.