Review main Merged

36556[FEATURE] Get a file's text content if possible

Ingo Renner CR +2 V +2 Hard +416 −0 updated 03/13/15, 08:42

Commit message

Currently FAL only allows to extract meta data from files. This
patch allows to also extract text content from files. This can
be useful for search engines or providing snippets/teasers
in document archives.

Multiple text extractors can be registered to allow dealing with
different file types. A plain text extractor is provided by the core.

This is also a successor to the former textExtract service interface
implemented by several extensions: http://bit.ly/1D0x92M

Review

  • Code review CR +2
    • Stefan Froemken +2
    • Frank Nägler +1
    • Frans Saris +1
  • Verified V +2
    • Stefan Froemken +2
    • Frank Nägler +1
    • Frans Saris +1

Files 7

Review log 35

Show 10 earlier messages
  1. Ingo Renner uploaded patch set 1 02/01/15, 06:58

  2. Ingo Renner uploaded patch set 2 02/01/15, 07:23

  3. Ingo Renner commented on patch set 2 02/01/15, 07:27

    Here's a version of EXT:tika that implements the interface https://github.com/irnnr/typo3-ext-tika
  4. Ingo Renner uploaded patch set 3 02/01/15, 07:36

  5. Stefan Froemken commented on patch set 3 02/01/15, 08:18

    Cool idea. I only miss one thing: You write the content into a PHP Variable and not directly to a database, search engine, what ever. So the whole content will be temporary saved in memory_limit of PHP. What do you think? Would it be good to test expected size against free PHP memory before extracting the content?
  6. Ingo Renner commented on patch set 3 02/01/15, 09:01

    @Stefan yes, the content is not written to the DB on purpose. Imagine a large pdf document library. You would not want to duplicate the content and thus a large amount of data. Most of the time this is simply not necessary. The client knows best what it wants to do with the text content and should take care of it. If we take the example of a snippet/teaser then such an extension could add a DB field. A search engine obviously will need the whole content, but they are pretty good at compressing it. So far (with previous EXT:solr text extraction) I never ran into a situation where we exceeded the PHP memory limit with extracted text.
  7. Frans Saris commented on patch set 3 CR -1 V -1 02/01/15, 09:25

    IMO the extractor stuff doesn't belong in the File object. Could you go for the same approach as is done for retrieving mimetype and file dimensions. See https://review.typo3.org/#/c/31256/
  8. Ingo Renner commented on patch set 3 02/01/15, 09:51

    Yeah I would agree with that. It didn't feel quite right, but since it's not that much I simply put it into the file object. I don't expect there to ever be a lot of text extractors. Could you go a bit more into detail what you want to move where?
  9. Frans Saris commented on patch set 3 02/01/15, 09:59

    Just add your new introduced method to FileInfo. Then someone who need the readable content of a file can use FileInfo to retrieve the text. We use that now also to retrieve file dimensions see https://review.typo3.org/#/c/33990/8/typo3/sysext/core/Classes/Resource/Index/MetaDataRepository.php. Here Fabien created a ImageInfo object that extends FileInfo. As this is filetype dependent it doesn't belong into the File object.
  10. Ingo Renner uploaded patch set 4 02/01/15, 10:16

  1. Michael Oehlhof voted on patch set 4 CR +1 02/01/15, 10:48

  2. Frans Saris commented on patch set 4 02/01/15, 11:17

    FileInfo.php:112

    This needs a Registry class just like there is for FAL drivers, collections, extractors ect
  3. Ingo Renner commented on patch set 4 02/01/15, 20:50

    Usually I would agree, but I think that would be too much overhead here. I don't expect there to be a lot of text extractors anyway. So I'd argue to keep it simple.
  4. Frans Saris commented on patch set 4 02/02/15, 09:51

    Hi Ingo, I pointed you into the wrong direction. 4 days of continuous coding was a little too much I guess :) The extractor shouldn't be part of the FileInfo as this is only for local file stuff (lowlevel, not fal related). Please have a look at the Renderer functionality https://review.typo3.org/#/c/32925/. This would also be the way to go for this. I guess you could just copy the RendererRegistry (including UnitTests) and rename it to TextExtractorRegistry. The benefit of adding a API for the registration of the textExtractors is that we are able to move the configuration storage with the next versions without breaking existing extractors. Also would it be easier to find for developers how to register their own stuff. And it isn't that much more work to implement it as you can copy most of the RendererRegistry :) Core would then ship a PlainTextExtractor class. Using the registry the get the text from a file would then look like: $textExtractorService =TextExtractorRegistry::getInstance()->getTextExtractorService($file); if ($textExtractorService !== NULL) { $text = $textExtractorService->extractText($file); } The benefit of this approach is also that you know if a textExtractor was found. Please let me know if you need any assistance with this or if you got questions about this. BTW a new feature also needs a rst file that describes the new functionality (see the patch linked patch for a example).
  5. Stefan Neufeind commented on patch set 4 02/03/15, 10:21

    Needs a rebase. Could somebody go at this shortly, so we might have it before feature-freeze?
  6. Frans Saris commented on patch set 4 02/03/15, 10:25

    This needs a rewrite not only a rebase. Ingo in working on that.
  7. Frans Saris voted on patch set 4 CR -1 02/03/15, 10:26

  8. Ingo Renner commented on patch set 4 02/03/15, 17:59

    Yeah, I'm on it. Will push later today.
  9. Ingo Renner uploaded patch set 5 02/08/15, 05:20

  10. Michael Oehlhof commented on patch set 5 02/14/15, 21:24

    Some formatting issues for the documentation.

    Feature-36743-FAL-TextExtractorRegistry.rst:37

    Should be .. code-block:: php instead of :: and indent the code block by 1 tab character. Remove .. at the end.

    Feature-36743-FAL-TextExtractorRegistry.rst:46

    Same as previuos code block.

    Feature-36743-FAL-TextExtractorRegistry.rst:54

    Should be 2 empty lines.
  11. Michael Oehlhof voted on patch set 5 CR +1 02/14/15, 21:57

  12. Ingo Renner uploaded patch set 6 02/19/15, 23:29

  13. Ingo Renner commented on patch set 5 02/19/15, 23:30

    Feature-36743-FAL-TextExtractorRegistry.rst:37 Reply

    Done

    Feature-36743-FAL-TextExtractorRegistry.rst:46 Reply

    Done

    Feature-36743-FAL-TextExtractorRegistry.rst:54 Reply

    Done
  14. Ingo Renner uploaded patch set 7 02/19/15, 23:33

  15. Michael Oehlhof voted on patch set 7 CR +1 02/20/15, 16:45

  16. Frank Nägler commented on patch set 7 CR +1 03/12/15, 18:17

    thx Ingo, nice patch and a great addition. I would like to see a final review by Frans.
  17. Frans Saris voted on patch set 7 CR +1 03/12/15, 18:43

  18. Frans Saris commented on patch set 7 03/12/15, 18:44

    If not merged yet I will test it tomorrow
  19. Stefan Froemken commented on patch set 7 V +1 03/12/15, 20:12

    I have created my own little extension to test that. Works good.
  20. Frans Saris commented on patch set 7 CR 0 03/13/15, 07:51

    Feature-36743-FAL-TextExtractorRegistry.rst:52

    if ($extractor !== NULL) { would be better advice :) as it is in line with our code style. Will push new patch
  21. Frans Saris uploaded patch set 8 03/13/15, 07:53

  22. Frans Saris voted on patch set 8 CR +1 V +1 03/13/15, 07:53

  23. Frank Nägler voted on patch set 8 V +1 CR +1 03/13/15, 07:55

  24. Stefan Froemken voted on patch set 8 CR +2 V +2 03/13/15, 08:42

  25. Stefan Froemken merged patch set 8 03/13/15, 08:42