Review main Merged

89987[TASK] Add more common MIME type replacements and file extensions

Garvin Hicking CR +2 V +2 Easy +32 −1 updated 09/08/25, 17:30

Commit message

MIME type recognition is not easy due to differences in Apache
vs. IANA defaults. Our mime-db integration has some mismatches
with what PHP's FileInfo or local OSes resolve by default.

To make this less painful, we manage our list in:

```
$GLOBALS['TYPO3_CONF_VARS']['SYS']['FileInfo']['mimeTypeCompatibility']
```

With this patch, we add more common types along the way:

* .srt (Video subtitles)
* .json
* .exe (not allowed by default, though)

This patch only deals with MIME type detection. #107398 and #107399
are follow-ups to integrate this with `textfile_ext`, `mediafile_ext`
and `miscfile_ext`.

Review

  • Code review CR +2 Done
    • Garvin Hicking +2
    • Georg Ringer +1
    • Jörg Bösche +1
    • Simon Schaufelberger +1
  • Verified V +2 Done
    • Garvin Hicking +2
    • Georg Ringer +1
    • Simon Schaufelberger +1
    • core-ci +1 CI

Issues 6

Files 1

Review log 3137

Show 912 earlier messages
  1. Garvin Hicking uploaded patch set 1 07/04/25, 08:28

  2. Garvin Hicking commented on patch set 1 07/04/25, 08:30

    Another approach would be to evaluate https://packagist.org/packages/fileeye/mimemap or other libs/packages, as was mentioned by Chris Wolff in slack.
  3. Mathias Brodala commented on patch set 1 07/04/25, 08:30

    DefaultConfiguration.php:442

    Tiny nitpick: we should keep this list sorted alphabetically by key.
  4. Garvin Hicking commented on patch set 1 07/04/25, 08:32

    DefaultConfiguration.php:442 Reply

    Yeah good idea, feel free to push over this to adjust. The whole section above is also not properly sorted.
  5. core-ci voted on patch set 1 V +1 07/04/25, 08:35 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/81315

  6. Benjamin Franzke marked it work in progress 07/04/25, 08:43

  7. Mathias Brodala commented on patch set 2 V +1 07/04/25, 10:30

    Published edit on patch set 1.
  8. Mathias Brodala commented on patch set 2 07/04/25, 10:30

    DefaultConfiguration.php:442 Reply

    Done
  9. core-ci voted on patch set 2 V +1 07/04/25, 10:37 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/81319

  10. Garvin Hicking commented on patch set 3 V +1 07/07/25, 09:00

    Published edit on patch set 2.
  11. core-ci voted on patch set 3 V +1 07/07/25, 09:08 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/81399

  12. Stephan Großberndt voted on patch set 3 CR +1 07/08/25, 09:38

  1. Garvin Hicking uploaded patch set 4 CR +1 V +1 08/01/25, 18:46

  2. core-ci voted on patch set 4 V +1 08/01/25, 18:54 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/82645

  3. Oliver Hader commented on patch set 4 08/03/25, 20:15

    DefaultConfiguration.php:110

    Can `rtf` be edited in the backend's text-editor? The others looks good...

    DefaultConfiguration.php:111

    Not sure about `srt`... parts of `SYS/mediafile_ext` should be kept in sync with `GFX/imagefile_ext`.

    DefaultConfiguration.php:112

    Not sure about all the macro-related files (what are the chances that those were downloaded and executed on users' devices automatically - I guess that's similar to `exe` binaries).

    DefaultConfiguration.php:438

    Isn't `application/json` implicitly available through `MimeTypeCompatibilityTypeGuesser`?
  4. Garvin Hicking commented on patch set 4 08/03/25, 20:37

    DefaultConfiguration.php:110 Reply

    I haven't checked how the editor could handle it, but RTF is ASCII so it should be plaintext editable - however some backlash-chars might make trouble

    DefaultConfiguration.php:111 Reply

    So better put srt into miscfile?

    DefaultConfiguration.php:112 Reply

    I mainly tried to think of which files editors in our instances want to "share" on their websites and offer for download. But our project's usages might not be a reasonable default. We can strip them, I'm not fighting for those (but could live with them ;)).

    DefaultConfiguration.php:438 Reply

    will need to check, I believe I was not able to upload a json file otherwise.
  5. Garvin Hicking uploaded patch set 5 V +1 08/18/25, 09:05

  6. Garvin Hicking commented on patch set 5 08/18/25, 09:06

    This change is ready for review.

    Reply

    (beyond scope for now)

    DefaultConfiguration.php:111 Reply

    Put it into textfile as well, can be edited easily.

    DefaultConfiguration.php:112 Reply

    I think websites might want to offer these files, especially when TYPO3 is used in Intranets which is a common use case, so for now I let it in. Agreed?

    DefaultConfiguration.php:438 Reply

    Only works with json explicitly mentioned. Also adding it to "text" has the advantage of the file being editable.
  7. core-ci voted on patch set 5 V +1 08/18/25, 09:13 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/83561

  8. Stefan Froemken commented on patch set 5 08/29/25, 09:49

    I have create a new file stefan.vcard and entered some text. In EXT:reports module I get following error now: Inconsistent files have been found Storage "fileadmin" (id:1) 1:/Template/stefan.vcard Mime-type "application/x-empty" not allowed for file extension "vcard" (expected: text/vcard). If I try to rename stefan.vcard to stefan.vcf I get following error: An error occurred Resource consistency check failed: Mime-type "application/x-empty" not allowed for file extension "vcf" (expected: text/vcard, text/x-vcard). In MimeType.php I only found a mapping from vcf to text/x-vcard, maybe that's why TYPO3 has problems identifiying files with vcard extension?!?!

    DefaultConfiguration.php:110

    Ahh...sorted. THX. Much better.
  9. Oliver Hader commented on patch set 5 08/29/25, 18:53

    Reply

    I guess this is still an issue (the comment was marked "resolved"). How did you modify the content of the file? I'm wondering about `application/x-empty`, which indicates the file is empty...
  10. Stefan Froemken commented on patch set 5 08/29/25, 20:21

    Reply

    I have created the file with file-list module -> Create new file. As it is not possible to edit the content of vcard files, only the file itself (empty) was created. That's why it was registered x-empty in sys_file. After adding some content via PhpStorm I get the inconsistency problem in reports module. Here a small video: https://drive.google.com/file/d/1VBsgu0jeBnlRsXKJ8DOt1uo2V_9EAzcq/view?usp=drive_link
  11. Garvin Hicking commented on patch set 5 08/29/25, 20:39

    Reply

    I believe I understand what happened here. Sorry for the deep-dive, but I'm hoping to clear things up why I believe this patch is right. Read it as a bedtime story. ;) Generally, the MIME consistency service works like this. A file's MIME type is does not exist inside a file, it is applied from "outside". This is done by either PHP's 'finfo' extension, or the 'mime_content_type' function. This utilizes either the so-called "magic.mime" file of the OS (mime_content_type) or libmagic directly (finfo_file). In both cases, a binary "magic" file is evaluated, that contains a huuuuge list of file inspections with several rules. For example, if a file begins with a JPEG/PNG byte header, it is quite easily mapped to this image format. For plaintext files it is much harder. HTML and PHP can also be well-detected (through their <html> or <?php tags). But other formats like vcard/vcf are harder, because their structure could also occur inside other regular files. That's kind of why it's "magic". It may fail and identify one file as a different format, just because of its markup. So that's just one problem MIME detection has (even though practically, the rules are pretty well battle-tasted by now). The other issue is that there is no really single authoritative MIME type registry. Both Apache and IANA try to define MIME types for specific file extensions. But sometimes they use different MIME types for the same "kind of file". TYPO3 uses PHPs MIME detection functionality, which is OS-dependant. Addtionally, it uses an independent MIME type collection library (NPM mime-db) that holds a mapping of file-extensions to MIME-types (and vice versa), and which tries to combine IANA and Apache "standards". Sadly, these sometimes mismatch. Example: * PHP reports a "file.vcf" as MIME type "text/vcard" (IANA "notation"). * NPM mime-db reports a "file.vcf" as "text/x-vcard" (Apache "notation"). * Funny side note: "file.vcard" is actually reported as "text/vcard" in mime-db The logic basically is this: * TYPO3 uses `Build/Scripts/generateMimeTypes.php` to parse the `Build/node_modules/mime-db/db.json` file into a PHP file saved as `typo3/sysext/core/Classes/Resource/MimeTypeCollection.php` * Some TYPO3 logic is already applied to make some adjustments to the MIME / file extension map. * This `MimeTypeCollection` is used in the MIME type consistency service to check if an uploaded (or existing file) matches the PHP-reported MIME-type with the mime-db MIME type related to its file extension. * This authoritative `MimeTypeCollection` map is expanded by the config option `[SYS][FileInfo][mimeTypeCompatibility]`. That array can be influenced by system maintainers, and is also maintained by the core as a default via `typo3/sysext/core/Configuration/DefaultConfiguration.php` * This happens through the `MimeTypeCompatibilityTypeGuesser` in `typo3/sysext/core/Classes/Resource/MimeTypeCompatibilityTypeGuesser.php`. This also does some wildcard yaml/json/xml assignment. * This is basically a map that says "this IANA MIME type is the same as this Apache MIME type", and also bundles several file extensions under "well known" MIME types. * As a last resort there's also a configuration `[SYS][FileInfo][fileExtensionToMimeType]` but that is beyond scope now, and should not be utilized. With that explained, what happened for Stefan was likely this: * He created a new empty file in `fileadmin` FAL storage called `test.vcard` * This lead to the `sys_file` entry to have a `mime_type` of "application/x-empty". Because the new file was initially empty. * Now the file was manually filled with content that MIME detection magic could identify as an actual VCard * After this, the "MIME consistency" report reveals a mismatch of "application/x-empty" not being the proper "text/vcard" MIME type in the database * To remedy this, one will needd to delete the wrong sys_file entry and re-add the file with the proper contents, so that MIME magic will detect it properly. The `Update "mime_type" field to extension specific types` wizard could be re-executed in this case. So: The consistency report in this case is a right one. The changes in this patch "only" add a few more MIME alternatives where IANA and Apache differ from each other to the "compatibility" map. It doesn't affect functionality that is already implemented. The other change is, that some common file extensions are also added to the list of being uploadable and text-editable (note: creation via "create new textfile" is independent from this!) What is IMO debatable: Should the backend's "create new textfile" functionality allow to create an empty "stefan.vcard" file? It lists textfiles that can be edited, but actually any filename can be used here - I can create an empty "image.jpg" file (MIME-Type "application/x-empty"). This is beyond the scope of this patch, but of course creates an inconsistency, if such file is edited/replaced. (If that is understandable and agreeable, I would be happy if we can resolve this thread)
  12. Stefan Froemken commented on patch set 5 CR +1 V +1 08/29/25, 21:04

    I exchanged some slack messages with Garvin. Thank you for your nice comments there and the explaining comment here at Gerrit. As I told you, it seems that this patch has nothing to do with my findings and thank you for that confirmation. You got my GO for this patch, but yes, the behavior with empty files should be solved somehow. But that is not part of this patch.
  13. Garvin Hicking commented on patch set 5 09/03/25, 15:15

    Reply

    Done
  14. Georg Ringer commented on patch set 5 V -1 09/03/25, 16:29

    I am not sure if we really want to pollute the list of file extensions in content elements like text with media or many custom extensions? furthermore it could be breaking if e.g. the template is adopted and can only cope with the previous list. this needs at least a breaking rst with information how to reduce it again feel free to reject my -1
  15. Garvin Hicking commented on patch set 5 09/03/25, 18:32

    Reply

    Thanks for the input. IMO this would be mainly a visual thing, or can you construct a use case where added file types would create trouble on backend uploads? The problem I see is that the security fix now might restrict files that people were used to being uploadable before. Maybe an important.rst instead of breaking.rst would do? Otherwise we could split up this patch in two parts: mimetypes as one, and enhancing plus sorting the _ext lists as another. The first half should be unproblematic then. Or we could just split off the mediafile_ext part because I think that is the only format I imagine used with f:media in the frontend that could behave differently with file types gertinf uploaded that were not uploaded before (but might have been already before the security fix as legitimate files). Happy to hear thoughts, I just want editor filemanagement handling be less impacted by the security additions for "known" files.
  16. Georg Ringer commented on patch set 5 09/04/25, 05:47

    Reply

    yeah, splitting would be a good idea IMHO! and important.rst as well!
  17. Garvin Hicking uploaded patch set 6 CR +1 V +1 09/05/25, 09:41

  18. core-ci voted on patch set 6 V +1 09/05/25, 09:49 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/84448

  19. Garvin Hicking commented on patch set 6 09/05/25, 10:21

    Reply

    Done, I've split this up in: 1. Address MIME types (this patch, v12-main) 2. Address text/miscfile (https://review.typo3.org/c/Packages/TYPO3.CMS/+/90575, v12-main) 3. Address mediafile (https://review.typo3.org/c/Packages/TYPO3.CMS/+/90577, main)
  20. Garvin Hicking commented on patch set 6 09/05/25, 10:21

    DefaultConfiguration.php:112 Reply

    Done
  21. Georg Ringer voted on patch set 6 CR +1 V +1 09/05/25, 14:53

  22. Simon Schaufelberger voted on patch set 6 CR +1 V +1 09/05/25, 18:41

  23. Jörg Bösche voted on patch set 6 CR +1 09/07/25, 18:35

  24. Garvin Hicking voted on patch set 6 CR +2 V +2 09/08/25, 17:17

  25. Garvin Hicking merged patch set 7 09/08/25, 17:30