96039[!!!][FEATURE] Replace indexed_search parsers by text extractors
Commit message
Review
-
Code review CR +2 Done
- Benni Mack +2
- Oli Bartsch +1
- Garvin Hicking +1
-
Verified V +2 Done
- Benni Mack +2
- Oli Bartsch +1
- Garvin Hicking +1
Issues 1
Files 68
- typo3/sysext/core/Classes/DependencyInjection/TextExtractorPass.php+3 −2
- typo3/sysext/core/Classes/Resource/TextExtraction/ExtractedText.phpnew+32 −0
- typo3/sysext/core/Classes/Resource/TextExtraction/LocalFileAccess.phpnew+61 −0
- typo3/sysext/core/Classes/Resource/TextExtraction/PlainTextExtractor.php6+16 −2
- typo3/sysext/core/Classes/Resource/TextExtraction/StructuredTextExtractorInterface.phpnew+57 −0
- typo3/sysext/core/Classes/Resource/TextExtraction/TextExtractorRegistry.php+136 −5
- typo3/sysext/core/Configuration/Services.yaml+2 −2
- typo3/sysext/core/Documentation/Changelog/15.0/Breaking-110775-ExternalDocumentParsersOfIndexedSearchReplacedByTextExtractors.rst2new+207 −0
- typo3/sysext/core/Documentation/Changelog/15.0/Feature-110775-StructuredTextExtractionInTextExtractorRegistry.rstnew+112 −0
- typo3/sysext/core/Tests/Functional/Resource/TextExtraction/LocalFileAccessTest.phpnew+100 −0
- typo3/sysext/core/Tests/Unit/DependencyInjection/TextExtractorPassTest.php+27 −0
- typo3/sysext/core/Tests/Unit/Resource/TextExtraction/PlainTextExtractorTest.php+16 −0
- typo3/sysext/core/Tests/Unit/Resource/TextExtraction/TextExtractorRegistryTest.php+230 −0
- typo3/sysext/indexed_search/Classes/Controller/AdministrationController.php+0 −11
- typo3/sysext/indexed_search/Classes/Controller/SearchController.php2+15 −36
- typo3/sysext/indexed_search/Classes/Domain/Repository/AdministrationRepository.php+5 −8
- typo3/sysext/indexed_search/Classes/Domain/Repository/IndexSearchRepository.php+4 −22
- typo3/sysext/indexed_search/Classes/Hook/DeleteIndexedData.php+8 −5
- typo3/sysext/indexed_search/Classes/Service/ExternalMediaTypes.phpnew+134 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/AbstractTextExtractor.php2new+175 −0
Show 48 more files
- typo3/sysext/indexed_search/Classes/TextExtraction/CatdocTextExtractor.phpnew+48 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/CsvTextExtractor.phpnew+50 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/ExternalCommandExecutor.phpnew+38 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/ExternalToolConfiguration.phpnew+87 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/HtmlContentConverter.phpnew+135 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/HtmlTextExtractor.phpnew+50 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/ImageCommentTextExtractor.phpnew+46 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/LegacyExcelTextExtractor.phpnew+58 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/LegacyPowerPointTextExtractor.phpnew+58 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/OfficeOpenXmlTextExtractor.phpnew+118 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/OpenDocumentTextExtractor.phpnew+88 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/PartitionedDocument.phpnew+47 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/PartitionedTextExtractorInterface.phpnew+43 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/PdfTextExtractor.phpnew+155 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/RtfTextExtractor.phpnew+57 −0
- typo3/sysext/indexed_search/Classes/TextExtraction/XmlTextExtractor.phpnew+60 −0
- typo3/sysext/indexed_search/Classes/FileContentParser.phpdeleted+0 −782
- typo3/sysext/indexed_search/Classes/Indexer.php4+185 −276
- typo3/sysext/indexed_search/Configuration/Services.yaml+4 −0
- typo3/sysext/indexed_search/Documentation/Introduction/Index.rst+3 −2
- typo3/sysext/indexed_search/Documentation/TechnicalDetails/ExternalMedia/Index.rst+4 −2
- typo3/sysext/indexed_search/Resources/Private/Language/locallang_em.xlf+2 −2
- typo3/sysext/indexed_search/Resources/Private/Language/locallang_main.xlf+14 −14
- typo3/sysext/indexed_search/Tests/Functional/Controller/SearchControllerTest.php+47 −1
- typo3/sysext/indexed_search/Tests/Functional/Fixtures/Files/document.docxnewbinary
- typo3/sysext/indexed_search/Tests/Functional/Fixtures/Files/outside.txtnew+1 −0
- typo3/sysext/indexed_search/Tests/Functional/Fixtures/Files/public.txtnew+1 −0
- typo3/sysext/indexed_search/Tests/Functional/Fixtures/Indexer/docx_with_container_mime_type.csvnew+3 −0
- typo3/sysext/indexed_search/Tests/Functional/Fixtures/Language/locallang_media_types.xlfnew+11 −0
- typo3/sysext/indexed_search/Tests/Functional/Hook/DeleteIndexedDataTest.phpnew+57 −0
- typo3/sysext/indexed_search/Tests/Functional/TextExtraction/TextExtractorRegistrationTest.phpnew+38 −0
- typo3/sysext/indexed_search/Tests/Functional/FileIndexingTest.phpnew+257 −0
- typo3/sysext/indexed_search/Tests/Functional/IndexSearchRepositoryTest.php+2 −2
- typo3/sysext/indexed_search/Tests/Functional/NonPublicStorageIndexingTest.php+0 −13
- typo3/sysext/indexed_search/Tests/Unit/Service/ExternalMediaTypesTest.phpnew+150 −0
- typo3/sysext/indexed_search/Tests/Unit/TextExtraction/AbstractTextExtractorTestCase.phpnew+104 −0
- typo3/sysext/indexed_search/Tests/Unit/TextExtraction/ExternalToolConfigurationTest.phpnew+92 −0
- typo3/sysext/indexed_search/Tests/Unit/TextExtraction/ExternalToolTextExtractorsTest.phpnew+349 −0
- typo3/sysext/indexed_search/Tests/Unit/TextExtraction/HtmlContentConverterTest.phpnew+108 −0
- typo3/sysext/indexed_search/Tests/Unit/TextExtraction/PdfTextExtractorTest.phpnew+188 −0
- typo3/sysext/indexed_search/Tests/Unit/FileContentParserTest.phpdeleted+0 −63
- typo3/sysext/indexed_search/Tests/Unit/IndexerTest.php+5 −44
- typo3/sysext/indexed_search/ext_conf_template.txt+0 −3
- typo3/sysext/indexed_search/ext_localconf.php+0 −32
- typo3/sysext/install/Configuration/ExtensionScanner/Php/ArrayDimensionMatcher.php+5 −0
- typo3/sysext/install/Configuration/ExtensionScanner/Php/ClassNameMatcher.php+5 −0
- typo3/sysext/install/Configuration/ExtensionScanner/Php/MethodCallMatcher.php+42 −0
- typo3/sysext/install/Configuration/ExtensionScanner/Php/PropertyPublicMatcher.php+15 −0
Review log 3450
Show 1725 earlier messages
-
Benni Mack uploaded patch set 1 09/24/26, 15:23
-
core-ci voted on patch set 1 V -1 09/24/26, 16:24 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/109168
-
Benni Mack uploaded patch set 2
V -109/24/26, 17:36 -
Benni Mack marked it ready for review 09/24/26, 17:37
-
Benni Mack marked it work in progress 09/24/26, 17:37
-
core-ci voted on patch set 2 V +1 09/24/26, 17:44 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/109175
-
Benni Mack wrote 09/25/26, 18:43
Topic set to indexed-search -
Benni Mack uploaded patch set 3
V +110/06/26, 19:53 -
core-ci voted on patch set 3 V +1 10/07/26, 09:22 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110276
-
Benni Mack uploaded patch set 4
V +110/09/26, 12:09 -
core-ci voted on patch set 4 V -1 10/09/26, 12:22 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110600
-
Benni Mack rebased onto patch set 5
V -110/09/26, 14:52 -
core-ci voted on patch set 5 V +1 10/09/26, 15:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/110614
-
Benni Mack marked it ready for review 10/09/26, 16:25
-
Andreas Kienast commented on patch set 5 10/10/26, 20:51
PlainTextExtractor.php:47This reads bogus compared with getSupportedFileExtensions() -> ['txt']. PlainTextExtractor now supports only .txt files and everything that is text/plain, e.g. .log files.AbstractTextExtractor.php:156We have similar code in PlainTextExtractor already. Another finding discovered by Claude, which sounds reasonable: > This can delete the original file from the storage. For a local storage, getForLocalProcessing(false) does not return a copy but the absolute path of the original file (LocalDriver::getFileForLocalProcessing()). Whether that path is unlinked is decided only by comparing its basename with sys_file.name. As soon as the two differ, the indexer deletes the original. > > I verified this with a functional test: a sys_file record with identifier /user_upload/data.csv and name Data.csv (differing in case only). After CsvTextExtractor::extractText(), fileadmin/user_upload/data.csv is gone. With a matching name, the file survives. > > The same pattern has existed in the core PlainTextExtractor since 2015 (PlainTextExtractor.php line 64). This change copies it into the base class of all indexed_search extractors and routes indexing through PlainTextExtractor as well, so every indexed file type is now affected. > > Please address this in this change: > > (1) Replace the basename comparison by GeneralUtility::unlink_tempfile($localFilePath), which only removes files below typo3temp/ or var/, where the drivers put their temporary copies. I checked that the original file survives with this change, with both a matching and a differing name. > (2) Apply the same fix to PlainTextExtractor::extractText(). > (3) Add a functional test with a sys_file name that differs from the basename of the identifier, asserting that the original file still exists after extraction. -
Benni Mack commented on patch set 5 10/11/26, 08:04
PlainTextExtractor.php:47Replyyeah, I built it this way due to the recent mime-type detection logic that we had to deal with the recent security topics earlier this year. in that regard, we might consider switching to mime-type detection completely instead of file extensions, wdyt? -
Andreas Kienast commented on patch set 5 10/12/26, 05:50
PlainTextExtractor.php:47ReplyI think going mime type-only is a good way to go as long we use the same mechanism we already have in terms of overlapping them since some files have ambiguous mime types, depending on the environment. -
Benni Mack uploaded patch set 6
V +110/24/26, 10:49 -
core-ci voted on patch set 6 V +1 10/24/26, 10:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112414
-
Benni Mack commented on patch set 6 10/24/26, 11:05
PlainTextExtractor.php:47ReplyDoneAbstractTextExtractor.php:156ReplyDid both: extractors now select by MIME type only (getSupportedMimeTypes()), the difference is left to the existing mimeTypeCompatibility resolution FAL applies when indexing. The original-file deletion is fixed via a shared LocalFileAccess helper that only removes driver copies below var/transient/ — better than unlink_tempfile(), which still delete originals of a private storage in var/. -
Benni Mack uploaded patch set 7
V +110/24/26, 11:42 -
core-ci voted on patch set 7 V +1 10/24/26, 11:50 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112416
-
Benni Mack uploaded patch set 8
V +110/28/26, 10:05 -
core-ci voted on patch set 8 V +1 10/28/26, 10:17 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112698
-
Josua Vogel uploaded patch set 9
V +110/29/26, 07:43
-
core-ci voted on patch set 9 V +1 10/29/26, 08:46 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112783
-
Benni Mack uploaded patch set 10
V +110/29/26, 10:33 -
core-ci voted on patch set 10 V +1 10/29/26, 10:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112793
-
Oli Bartsch commented on patch set 10 10/29/26, 21:15
PlainTextExtractor.php:56Can we split it into its own bugfix for 14.3 as well?DefaultConfiguration.php:444Those also feed `ResourceConsistencyService` and `FileExtensionMimeTypeConsistencyValidator`. That relaxes the upload consistency check, and some mapped types (e.g. odg, otg, ..) have no extractor at all. Can we please move this to a separate bugifx with a backport, trimmed to the types really needed, and state the effect on upload validation?SearchController.php:766This returns an empty string for any extension not in `LABEL_KEYS`, so the `mediaTypes.<ext>` fallback never runs and file types of custom extractors never show up in the media selector.Indexer.php:391AFAICS nothing catches exceptions from extractors (including third-party ones) or from remote storage drivers, so one failing linked file can turn the first uncached page render into an error page.Indexer.php:670`extract()` and `indexFileDocument()` ask all structured extractors first and only then the legacy ones, while `getTextExtractor()` respects `#[AsTextExtractor]` priority. E.g. a legacy extractor with priority 50 for `application/pdf` is overridden by `PdfTextExtractor` in indexing but still returned by `getTextExtractor()`. Suggest to either deprecate `TextExtractorInterface`/`getTextExtractor()` for v16 (with Deprecation RST and scanner entry) or select in one loop over all instances in priority order, and add a test with both kinds for the same MIME type. -
Benni Mack uploaded patch set 11
V +110/30/26, 09:11 -
Benni Mack commented on patch set 11 10/30/26, 09:12
PlainTextExtractor.php:56ReplySearchController.php:766ReplyDone -
core-ci voted on patch set 11 V -1 10/30/26, 09:27 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112838
-
Benni Mack commented on patch set 11 10/30/26, 09:47
DefaultConfiguration.php:444Reply -
Benni Mack commented on patch set 11 10/30/26, 09:48
Indexer.php:391ReplyDoneIndexer.php:670ReplyI went with your second option: one selection in priority order. The new TextExtractorRegistry::findTextExtractor() asks all extractors in #[AsTextExtractor] priority order: - A structured extractor matches by the MIME type of the resource. - A legacy one matches by canExtractText(). The mimeTypeCompatibility mapping is only tried as a second pass if nothing matched. extract() and the Indexer both use it. I didn't deprecate TextExtractorInterface, because both interfaces can live side by side without conflict now. If you still prefer deprecating it for v16, I'd rather do that as a follow-up. -
core-ci voted on patch set 11 V +1 10/30/26, 11:00 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112838
-
Oli Bartsch uploaded patch set 12
V +110/30/26, 18:57 -
Oli Bartsch commented on patch set 12 CR +1 V +1 10/30/26, 18:59
Hey Benni, I just did a quick last update for the patch. The registry now has one way to pick an extractor: getStructuredTextExtractor() is gone, its mimeTypeCompatibility fallback is part of findTextExtractor(), and getSupportedMimeTypes() is protected. The tests and the Feature RST were updated to match. The Indexer no longer looks up every link in FAL: extractHyperLinks() only returns tag and href. extractLinks() resolves public paths only if their extension can be indexed and is not ignored, so unrelated files no longer get sys_file records. Additionally, the Breaking RST no longer suggests mapping application/octet-stream (it warns that this would weaken upload validation) and now explains why URL, jumpurl and non-FAL indexing were dropped. The Feature RST now says which extractor interface to implement and why both stay in v15, rather than deprecating TextExtractorInterface now. -
core-ci voted on patch set 12 V +1 10/30/26, 19:09 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112903
-
Garvin Hicking commented on patch set 12 10/30/26, 19:37
Breaking-110775-ExternalDocumentParsersOfIndexedSearchReplacedByTextExtractors.rst:14we can use :composer:`...` for this, but we'll most likely address this in the v15 release changelog rework. But maybe brief agents / dev-companion that we have this :) (Also, no double backticks in ReST files) - already pinged benji -
Garvin Hicking commented on patch set 12 10/30/26, 19:42
Notes: - `TYPO3SEARCH_begin/end` will no longer be parsed in HTML files. Is this intentional? Then please mention in the breaking ReST. - `sxc/sxi/sxw` handling no longer covers a mis-reported `application/zip` fallback of older libmagic instances. I think that's fine for these old formats, but maybe "worth pointing out" - The tool-specific diagnostics are effectively gone. When a tool is missing, `getSupportedMimeTypes()` returns `[]` (AbstractTextExtractor.php:67). The registry then never selects that extractor, so the log only says "No text extractor found". The old messages (`pdfToolsNotFound, catdocDisabled, …`), which named the wrong path, are marked unused. The notice in `checkRequiredTools()` (AbstractTextExtractor.php:119) is practically never reached from the indexer, which leaves admins with less to debug from. Maybe this isn't very important and is info enough?! -
Benni Mack uploaded patch set 13
CR +1V +110/30/26, 21:11 -
Oli Bartsch uploaded patch set 14 10/30/26, 21:14
-
Oli Bartsch uploaded patch set 15 10/30/26, 21:18
-
Oli Bartsch commented on patch set 15 CR +1 V +1 10/30/26, 21:18
PS14 was an accidental overwrite: I pushed my local fixes for Garvin's notes on top of PS12 without seeing PS13. PS15 restores PS13 unchanged, it already covers all of Garvin's points. 👍 -
Garvin Hicking commented on patch set 15 CR +1 V +1 10/30/26, 21:21
Reply
DoneGood one! 😊Breaking-110775-ExternalDocumentParsersOfIndexedSearchReplacedByTextExtractors.rst:14ReplyDone -
core-ci voted on patch set 13 V +1 10/30/26, 21:23 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112908
-
Benni Mack voted on patch set 15 CR +2 V +2 10/30/26, 21:23
-
Benni Mack merged patch set 16 10/30/26, 21:23
-
core-ci voted on patch set 14 V +1 10/30/26, 21:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112910
-
core-ci voted on patch set 15 V +1 10/30/26, 21:30 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/112912