Review main Merged Breaking

96039[!!!][FEATURE] Replace indexed_search parsers by text extractors

Benni Mack CR +2 V +2 Nightmare +4170 −1329 updated 10/30/26, 21:30

Commit message

The external document parsers of EXT:indexed_search were duck-typed
classes registered in $GLOBALS['TYPO3_CONF_VARS']['EXTCONF'] and
bound to the indexer by a back-reference. They are replaced by text
extractors, registered as tagged services and resolved through the
TextExtractorRegistry of EXT:core.

This makes file text extraction a Core API that any consumer can use,
not only indexed_search. The API is extended additively: the new
StructuredTextExtractorInterface works on any system resource, so
FAL files as well as files of packages or public files outside of a
storage, and returns an ExtractedText with title, description and
keywords next to the text. It declares the MIME types it supports,
so consumers can list file types without a file at hand, and the
registry selects extractors by MIME type, not by file extension.
Files already in the FAL index with a generic MIME type are mapped
by their file extension through "mimeTypeCompatibility".
TextExtractorInterface, getTextExtractor() and #[AsTextExtractor]
stay unchanged, an extractor may implement both interfaces. The new
findTextExtractor() asks extractors of both interfaces in the order
of their priority, extract() and the indexer use it.

Extractors read resources through a shared helper. It provides other
system resources as a temporary copy of their contents, and removes
only temporary copies below var/transient/.

EXT:indexed_search ships one extractor per document family, PDF page
ranges use a dedicated partitioned interface, and a new service
provides media type labels and icons for the search form and the
backend module. The "mediaTypes.<extension>" label of the plugin
names the file types of custom extractors in the search form. An
exception thrown while a linked file is indexed is logged and the
file is skipped, so it does not break the rendering of the page.
A link to a public file is only resolved through FAL if its file
extension can be indexed, so other linked files are not added to
the FAL index.

Only files of a FAL storage are indexed now. Indexing external URLs,
following jumpurl links and absolute file system paths is dropped.
The indexer handles every linked document as a FAL file: the
extractor is selected by the stored MIME type, time and size are
read from the FAL index, and access to non-public storages is
checked by the file dump token. URLs and arbitrary paths have no
such record and would need a second indexing path with its own HTTP
client and temporary files. JumpURLs are not generated by TYPO3 Core
since TYPO3 v7.5.

Review

  • Code review CR +2 Done
    • Benni Mack +2
    • Oli Bartsch +1
    • Garvin Hicking +1
  • Verified V +2 Done
    • Benni Mack +2
    • Oli Bartsch +1
    • Garvin Hicking +1

Issues 1

Files 68

Show 48 more files

Review log 3450

Show 1725 earlier messages
  1. Benni Mack uploaded patch set 1 09/24/26, 15:23

  2. core-ci voted on patch set 1 V -1 09/24/26, 16:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/109168

  3. Benni Mack uploaded patch set 2 V -1 09/24/26, 17:36

  4. Benni Mack marked it ready for review 09/24/26, 17:37

  5. Benni Mack marked it work in progress 09/24/26, 17:37

  6. core-ci voted on patch set 2 V +1 09/24/26, 17:44 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/109175

  7. Benni Mack wrote 09/25/26, 18:43

    Topic set to indexed-search
  8. Benni Mack uploaded patch set 3 V +1 10/06/26, 19:53

  9. core-ci voted on patch set 3 V +1 10/07/26, 09:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110276

  10. Benni Mack uploaded patch set 4 V +1 10/09/26, 12:09

  11. core-ci voted on patch set 4 V -1 10/09/26, 12:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110600

  12. Benni Mack rebased onto patch set 5 V -1 10/09/26, 14:52

  13. core-ci voted on patch set 5 V +1 10/09/26, 15:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110614

  14. Benni Mack marked it ready for review 10/09/26, 16:25

  15. Andreas Kienast commented on patch set 5 10/10/26, 20:51

    PlainTextExtractor.php:47

    This reads bogus compared with getSupportedFileExtensions() -> ['txt']. PlainTextExtractor now supports only .txt files and everything that is text/plain, e.g. .log files.

    AbstractTextExtractor.php:156

    We have similar code in PlainTextExtractor already. Another finding discovered by Claude, which sounds reasonable: > This can delete the original file from the storage. For a local storage, getForLocalProcessing(false) does not return a copy but the absolute path of the original file (LocalDriver::getFileForLocalProcessing()). Whether that path is unlinked is decided only by comparing its basename with sys_file.name. As soon as the two differ, the indexer deletes the original. > > I verified this with a functional test: a sys_file record with identifier /user_upload/data.csv and name Data.csv (differing in case only). After CsvTextExtractor::extractText(), fileadmin/user_upload/data.csv is gone. With a matching name, the file survives. > > The same pattern has existed in the core PlainTextExtractor since 2015 (PlainTextExtractor.php line 64). This change copies it into the base class of all indexed_search extractors and routes indexing through PlainTextExtractor as well, so every indexed file type is now affected. > > Please address this in this change: > > (1) Replace the basename comparison by GeneralUtility::unlink_tempfile($localFilePath), which only removes files below typo3temp/ or var/, where the drivers put their temporary copies. I checked that the original file survives with this change, with both a matching and a differing name. > (2) Apply the same fix to PlainTextExtractor::extractText(). > (3) Add a functional test with a sys_file name that differs from the basename of the identifier, asserting that the original file still exists after extraction.
  16. Benni Mack commented on patch set 5 10/11/26, 08:04

    PlainTextExtractor.php:47 Reply

    yeah, I built it this way due to the recent mime-type detection logic that we had to deal with the recent security topics earlier this year. in that regard, we might consider switching to mime-type detection completely instead of file extensions, wdyt?
  17. Andreas Kienast commented on patch set 5 10/12/26, 05:50

    PlainTextExtractor.php:47 Reply

    I think going mime type-only is a good way to go as long we use the same mechanism we already have in terms of overlapping them since some files have ambiguous mime types, depending on the environment.
  18. Benni Mack uploaded patch set 6 V +1 10/24/26, 10:49

  19. core-ci voted on patch set 6 V +1 10/24/26, 10:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112414

  20. Benni Mack commented on patch set 6 10/24/26, 11:05

    PlainTextExtractor.php:47 Reply

    Done

    AbstractTextExtractor.php:156 Reply

    Did both: extractors now select by MIME type only (getSupportedMimeTypes()), the difference is left to the existing mimeTypeCompatibility resolution FAL applies when indexing. The original-file deletion is fixed via a shared LocalFileAccess helper that only removes driver copies below var/transient/ — better than unlink_tempfile(), which still delete originals of a private storage in var/.
  21. Benni Mack uploaded patch set 7 V +1 10/24/26, 11:42

  22. core-ci voted on patch set 7 V +1 10/24/26, 11:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112416

  23. Benni Mack uploaded patch set 8 V +1 10/28/26, 10:05

  24. core-ci voted on patch set 8 V +1 10/28/26, 10:17 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112698

  25. Josua Vogel uploaded patch set 9 V +1 10/29/26, 07:43

  1. core-ci voted on patch set 9 V +1 10/29/26, 08:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112783

  2. Benni Mack uploaded patch set 10 V +1 10/29/26, 10:33

  3. core-ci voted on patch set 10 V +1 10/29/26, 10:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112793

  4. Oli Bartsch commented on patch set 10 10/29/26, 21:15

    PlainTextExtractor.php:56

    Can we split it into its own bugfix for 14.3 as well?

    DefaultConfiguration.php:444

    Those also feed `ResourceConsistencyService` and `FileExtensionMimeTypeConsistencyValidator`. That relaxes the upload consistency check, and some mapped types (e.g. odg, otg, ..) have no extractor at all. Can we please move this to a separate bugifx with a backport, trimmed to the types really needed, and state the effect on upload validation?

    SearchController.php:766

    This returns an empty string for any extension not in `LABEL_KEYS`, so the `mediaTypes.<ext>` fallback never runs and file types of custom extractors never show up in the media selector.

    Indexer.php:391

    AFAICS nothing catches exceptions from extractors (including third-party ones) or from remote storage drivers, so one failing linked file can turn the first uncached page render into an error page.

    Indexer.php:670

    `extract()` and `indexFileDocument()` ask all structured extractors first and only then the legacy ones, while `getTextExtractor()` respects `#[AsTextExtractor]` priority. E.g. a legacy extractor with priority 50 for `application/pdf` is overridden by `PdfTextExtractor` in indexing but still returned by `getTextExtractor()`. Suggest to either deprecate `TextExtractorInterface`/`getTextExtractor()` for v16 (with Deprecation RST and scanner entry) or select in one loop over all instances in priority order, and add a test with both kinds for the same MIME type.
  5. Benni Mack uploaded patch set 11 V +1 10/30/26, 09:11

  6. Benni Mack commented on patch set 11 10/30/26, 09:12

    PlainTextExtractor.php:56 Reply

    SearchController.php:766 Reply

    Done
  7. core-ci voted on patch set 11 V -1 10/30/26, 09:27 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112838

  8. Benni Mack commented on patch set 11 10/30/26, 09:47

    DefaultConfiguration.php:444 Reply

  9. Benni Mack commented on patch set 11 10/30/26, 09:48

    Indexer.php:391 Reply

    Done

    Indexer.php:670 Reply

    I went with your second option: one selection in priority order. The new TextExtractorRegistry::findTextExtractor() asks all extractors in #[AsTextExtractor] priority order: - A structured extractor matches by the MIME type of the resource. - A legacy one matches by canExtractText(). The mimeTypeCompatibility mapping is only tried as a second pass if nothing matched. extract() and the Indexer both use it. I didn't deprecate TextExtractorInterface, because both interfaces can live side by side without conflict now. If you still prefer deprecating it for v16, I'd rather do that as a follow-up.
  10. core-ci voted on patch set 11 V +1 10/30/26, 11:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112838

  11. Oli Bartsch uploaded patch set 12 V +1 10/30/26, 18:57

  12. Oli Bartsch commented on patch set 12 CR +1 V +1 10/30/26, 18:59

    Hey Benni, I just did a quick last update for the patch. The registry now has one way to pick an extractor: getStructuredTextExtractor() is gone, its mimeTypeCompatibility fallback is part of findTextExtractor(), and getSupportedMimeTypes() is protected. The tests and the Feature RST were updated to match. The Indexer no longer looks up every link in FAL: extractHyperLinks() only returns tag and href. extractLinks() resolves public paths only if their extension can be indexed and is not ignored, so unrelated files no longer get sys_file records. Additionally, the Breaking RST no longer suggests mapping application/octet-stream (it warns that this would weaken upload validation) and now explains why URL, jumpurl and non-FAL indexing were dropped. The Feature RST now says which extractor interface to implement and why both stay in v15, rather than deprecating TextExtractorInterface now.
  13. core-ci voted on patch set 12 V +1 10/30/26, 19:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112903

  14. Garvin Hicking commented on patch set 12 10/30/26, 19:37

    Breaking-110775-ExternalDocumentParsersOfIndexedSearchReplacedByTextExtractors.rst:14

    we can use :composer:`...` for this, but we'll most likely address this in the v15 release changelog rework. But maybe brief agents / dev-companion that we have this :) (Also, no double backticks in ReST files) - already pinged benji
  15. Garvin Hicking commented on patch set 12 10/30/26, 19:42

    Notes: - `TYPO3SEARCH_begin/end` will no longer be parsed in HTML files. Is this intentional? Then please mention in the breaking ReST. - `sxc/sxi/sxw` handling no longer covers a mis-reported `application/zip` fallback of older libmagic instances. I think that's fine for these old formats, but maybe "worth pointing out" - The tool-specific diagnostics are effectively gone. When a tool is missing, `getSupportedMimeTypes()` returns `[]` (AbstractTextExtractor.php:67). The registry then never selects that extractor, so the log only says "No text extractor found". The old messages (`pdfToolsNotFound, catdocDisabled, …`), which named the wrong path, are marked unused. The notice in `checkRequiredTools()` (AbstractTextExtractor.php:119) is practically never reached from the indexer, which leaves admins with less to debug from. Maybe this isn't very important and is info enough?!
  16. Benni Mack uploaded patch set 13 CR +1 V +1 10/30/26, 21:11

  17. Oli Bartsch uploaded patch set 14 10/30/26, 21:14

  18. Oli Bartsch uploaded patch set 15 10/30/26, 21:18

  19. Oli Bartsch commented on patch set 15 CR +1 V +1 10/30/26, 21:18

    PS14 was an accidental overwrite: I pushed my local fixes for Garvin's notes on top of PS12 without seeing PS13. PS15 restores PS13 unchanged, it already covers all of Garvin's points. 👍
  20. Garvin Hicking commented on patch set 15 CR +1 V +1 10/30/26, 21:21

    Reply

    Done
    Good one! 😊

    Breaking-110775-ExternalDocumentParsersOfIndexedSearchReplacedByTextExtractors.rst:14 Reply

    Done
  21. core-ci voted on patch set 13 V +1 10/30/26, 21:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112908

  22. Benni Mack voted on patch set 15 CR +2 V +2 10/30/26, 21:23

  23. Benni Mack merged patch set 16 10/30/26, 21:23

  24. core-ci voted on patch set 14 V +1 10/30/26, 21:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112910

  25. core-ci voted on patch set 15 V +1 10/30/26, 21:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112912