36556[FEATURE] Get a file's text content if possible
Commit message
Review
-
Code review CR +2
- Stefan Froemken +2
- Frank Nägler +1
- Frans Saris +1
-
Verified V +2
- Stefan Froemken +2
- Frank Nägler +1
- Frans Saris +1
Files 7
- typo3/sysext/core/Classes/Resource/TextExtraction/PlainTextExtractor.phpnew+65 −0
- typo3/sysext/core/Classes/Resource/TextExtraction/TextExtractorInterface.phpnew+44 −0
- typo3/sysext/core/Classes/Resource/TextExtraction/TextExtractorRegistry.phpnew+113 −0
- typo3/sysext/core/Documentation/Changelog/master/Feature-36743-FAL-TextExtractorRegistry.rst7new+62 −0
- typo3/sysext/core/Tests/Unit/Resource/TextExtraction/PlainTextExtractorTest.phpnew+50 −0
- typo3/sysext/core/Tests/Unit/Resource/TextExtraction/TextExtractorRegistryTest.phpnew+79 −0
- typo3/sysext/core/ext_localconf.php+3 −0
Review log 35
Show 10 earlier messages
-
Ingo Renner uploaded patch set 1 02/01/15, 06:58
-
Ingo Renner uploaded patch set 2 02/01/15, 07:23
-
Ingo Renner commented on patch set 2 02/01/15, 07:27
Here's a version of EXT:tika that implements the interface https://github.com/irnnr/typo3-ext-tika -
Ingo Renner uploaded patch set 3 02/01/15, 07:36
-
Stefan Froemken commented on patch set 3 02/01/15, 08:18
Cool idea. I only miss one thing: You write the content into a PHP Variable and not directly to a database, search engine, what ever. So the whole content will be temporary saved in memory_limit of PHP. What do you think? Would it be good to test expected size against free PHP memory before extracting the content? -
Ingo Renner commented on patch set 3 02/01/15, 09:01
@Stefan yes, the content is not written to the DB on purpose. Imagine a large pdf document library. You would not want to duplicate the content and thus a large amount of data. Most of the time this is simply not necessary. The client knows best what it wants to do with the text content and should take care of it. If we take the example of a snippet/teaser then such an extension could add a DB field. A search engine obviously will need the whole content, but they are pretty good at compressing it. So far (with previous EXT:solr text extraction) I never ran into a situation where we exceeded the PHP memory limit with extracted text. -
Frans Saris commented on patch set 3 CR -1 V -1 02/01/15, 09:25
IMO the extractor stuff doesn't belong in the File object. Could you go for the same approach as is done for retrieving mimetype and file dimensions. See https://review.typo3.org/#/c/31256/ -
Ingo Renner commented on patch set 3 02/01/15, 09:51
Yeah I would agree with that. It didn't feel quite right, but since it's not that much I simply put it into the file object. I don't expect there to ever be a lot of text extractors. Could you go a bit more into detail what you want to move where? -
Frans Saris commented on patch set 3 02/01/15, 09:59
Just add your new introduced method to FileInfo. Then someone who need the readable content of a file can use FileInfo to retrieve the text. We use that now also to retrieve file dimensions see https://review.typo3.org/#/c/33990/8/typo3/sysext/core/Classes/Resource/Index/MetaDataRepository.php. Here Fabien created a ImageInfo object that extends FileInfo. As this is filetype dependent it doesn't belong into the File object. -
Ingo Renner uploaded patch set 4 02/01/15, 10:16
-
Michael Oehlhof voted on patch set 4 CR +1 02/01/15, 10:48
-
Frans Saris commented on patch set 4 02/01/15, 11:17
FileInfo.php:112This needs a Registry class just like there is for FAL drivers, collections, extractors ect -
Ingo Renner commented on patch set 4 02/01/15, 20:50
Usually I would agree, but I think that would be too much overhead here. I don't expect there to be a lot of text extractors anyway. So I'd argue to keep it simple. -
Frans Saris commented on patch set 4 02/02/15, 09:51
Hi Ingo, I pointed you into the wrong direction. 4 days of continuous coding was a little too much I guess :) The extractor shouldn't be part of the FileInfo as this is only for local file stuff (lowlevel, not fal related). Please have a look at the Renderer functionality https://review.typo3.org/#/c/32925/. This would also be the way to go for this. I guess you could just copy the RendererRegistry (including UnitTests) and rename it to TextExtractorRegistry. The benefit of adding a API for the registration of the textExtractors is that we are able to move the configuration storage with the next versions without breaking existing extractors. Also would it be easier to find for developers how to register their own stuff. And it isn't that much more work to implement it as you can copy most of the RendererRegistry :) Core would then ship a PlainTextExtractor class. Using the registry the get the text from a file would then look like: $textExtractorService =TextExtractorRegistry::getInstance()->getTextExtractorService($file); if ($textExtractorService !== NULL) { $text = $textExtractorService->extractText($file); } The benefit of this approach is also that you know if a textExtractor was found. Please let me know if you need any assistance with this or if you got questions about this. BTW a new feature also needs a rst file that describes the new functionality (see the patch linked patch for a example). -
Stefan Neufeind commented on patch set 4 02/03/15, 10:21
Needs a rebase. Could somebody go at this shortly, so we might have it before feature-freeze? -
Frans Saris commented on patch set 4 02/03/15, 10:25
This needs a rewrite not only a rebase. Ingo in working on that. -
Frans Saris voted on patch set 4 CR -1 02/03/15, 10:26
-
Ingo Renner commented on patch set 4 02/03/15, 17:59
Yeah, I'm on it. Will push later today. -
Ingo Renner uploaded patch set 5 02/08/15, 05:20
-
Michael Oehlhof commented on patch set 5 02/14/15, 21:24
Some formatting issues for the documentation.Feature-36743-FAL-TextExtractorRegistry.rst:37Should be .. code-block:: php instead of :: and indent the code block by 1 tab character. Remove .. at the end.Feature-36743-FAL-TextExtractorRegistry.rst:46Same as previuos code block.Feature-36743-FAL-TextExtractorRegistry.rst:54Should be 2 empty lines. -
Michael Oehlhof voted on patch set 5 CR +1 02/14/15, 21:57
-
Ingo Renner uploaded patch set 6 02/19/15, 23:29
-
Ingo Renner commented on patch set 5 02/19/15, 23:30
Feature-36743-FAL-TextExtractorRegistry.rst:37ReplyDoneFeature-36743-FAL-TextExtractorRegistry.rst:46ReplyDoneFeature-36743-FAL-TextExtractorRegistry.rst:54ReplyDone -
Ingo Renner uploaded patch set 7 02/19/15, 23:33
-
Michael Oehlhof voted on patch set 7 CR +1 02/20/15, 16:45
-
Frank Nägler commented on patch set 7 CR +1 03/12/15, 18:17
thx Ingo, nice patch and a great addition. I would like to see a final review by Frans. -
Frans Saris voted on patch set 7 CR +1 03/12/15, 18:43
-
Frans Saris commented on patch set 7 03/12/15, 18:44
If not merged yet I will test it tomorrow -
Stefan Froemken commented on patch set 7 V +1 03/12/15, 20:12
I have created my own little extension to test that. Works good. -
Frans Saris commented on patch set 7 CR 0 03/13/15, 07:51
Feature-36743-FAL-TextExtractorRegistry.rst:52if ($extractor !== NULL) { would be better advice :) as it is in line with our code style. Will push new patch -
Frans Saris uploaded patch set 8 03/13/15, 07:53
-
Frans Saris voted on patch set 8 CR +1 V +1 03/13/15, 07:53
-
Frank Nägler voted on patch set 8 V +1 CR +1 03/13/15, 07:55
-
Stefan Froemken voted on patch set 8 CR +2 V +2 03/13/15, 08:42
-
Stefan Froemken merged patch set 8 03/13/15, 08:42