Review main Merged

74742[BUGFIX] Prevent exception caused by hash collisions in indexed_search

Florian Schöppe CR +2 V +2 Normal +152 −2 updated 07/05/23, 17:18

Commit message

The indexed_search extension uses a truncated md5 hash for building a
word index. The truncation greatly increases the probability for
collisions. These collisions are the root cause for the problems that
are described in the related issues.

The best solution would be to either use full-length hashes or no
hashes at all. As this is not possible without changing the database
schema and rebuilding the whole word index, this patch prevents the
exceptions caused by the hash collisions. This is done by removing all
words with colliding hashes before the word index is updated.

Note that this patch just makes it possible to index page contents that
contain hash collisions (words with colliding hashes) again. The
affected words are obviously not added to the word index and though
won't yield results when searched for.

Review

  • Code review CR +2 Done
    • Stefan Bürk +2
    • Benni Mack +1
  • Verified V +2 Done
    • Stefan Bürk +2
    • Benni Mack +1
    • core-ci +1 CI

Issues 4

Files 5

Review log 3444

Show 1519 earlier messages
  1. Florian Schöppe uploaded patch set 1 05/25/22, 12:23

  2. core-ci voted on patch set 1 V +1 05/25/22, 12:31 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/27452

  3. Jonas Eberle commented on patch set 1 V +1 05/25/22, 14:10

    Thank you! It's easily reproducible with the wordlist from #84541. I've tested in a v10 and v11 and also watched out for performance regressions.

    Indexer.php:2046

    @param and @return PHP doc could be string[]
  4. Jonas Eberle commented on patch set 1 05/25/22, 14:16

    Additional note for testers: The error is only reproducible if EXTENSIONS/indexed_search/useMysqlFulltext is set.
  5. Jonas Eberle commented on patch set 1 05/25/22, 14:18

    Reply

    > Additional note for testers: The error is only reproducible if EXTENSIONS/indexed_search/useMysqlFulltext is set. Correction. If it is *not* set of course.
  6. Florian Schöppe uploaded patch set 2 05/25/22, 17:08

  7. Florian Schöppe commented on patch set 2 05/25/22, 17:10

    Indexer.php:2046 Reply

    Done
  8. core-ci voted on patch set 2 V -1 05/25/22, 17:16 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/27460

  9. Florian Schöppe uploaded patch set 3 05/25/22, 17:36

  10. Florian Schöppe commented on patch set 3 05/25/22, 17:39

    Indexer.php:2046 Reply

    Just reverted that change: The variable name $wordList suggests the array is an array of strings but in fact both, the argument and the returned array, have a more complex structure.
  11. core-ci voted on patch set 3 V +1 05/25/22, 17:45 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/27462

  12. Jonas Eberle commented on patch set 3 05/25/22, 17:47

    Sorry, my bad! That not just a string[].
  13. Jonas Eberle voted on patch set 3 V +1 05/25/22, 17:47

  14. crell commented on patch set 3 CR +1 05/27/22, 16:21

    One style comment, but otherwise seems fine.

    Indexer.php:1857

    Why not an array_filter() here, when you use one further down?
  15. Sybille Peters commented on patch set 3 V +1 05/28/22, 06:00

    Tested and verified that exception is not thrown with steps to reproduce from issue: https://forge.typo3.org/issues/84541#Reproduce
  16. Florian Schöppe commented on patch set 3 06/02/22, 06:26

    Indexer.php:1857 Reply

    You're right this is inconsistent. I kept the foreach/unset code style when refactoring the old code but introduced the array_filter with the new code. I think it would be best to replace the array_filter with foreach code because array_filter is rarely used throughout indexed_search (and it would be slightly better considering memory and performance).
  17. Florian Schöppe uploaded patch set 4 06/02/22, 06:40

  18. Florian Schöppe commented on patch set 4 06/02/22, 06:44

    Indexer.php:1857 Reply

    Done
  19. core-ci voted on patch set 4 V +1 06/02/22, 06:48 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/27643

  1. Jonas Eberle voted on patch set 4 V +1 CR +1 09/08/22, 11:31

  2. Anja Leichsenring commented on patch set 4 11/03/22, 19:16

    should we really backport that into v10? I would just say, main and 11.5 is enough.
  3. crell wrote 11/04/22, 00:22

    Removed reviewer crell.
  4. Benni Mack updated the commit message in patch set 5 CR +1 V +1 05/03/23, 09:25

  5. core-ci voted on patch set 5 V -1 05/03/23, 11:03 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/41409

  6. Sybille Peters wrote 05/03/23, 12:21

    Removed reviewer Sybille Peters.
  7. Jonas Eberle rebased onto patch set 6 V -1 06/27/23, 15:50

  8. Jonas Eberle commented on patch set 6 V +1 06/27/23, 15:50

    Just rebased.
  9. core-ci voted on patch set 6 V -1 06/27/23, 15:57 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43042

  10. Jonas Eberle uploaded patch set 7 V +1 06/29/23, 09:18

  11. core-ci voted on patch set 7 V +1 06/29/23, 09:25 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43104

  12. Jonas Eberle voted on patch set 7 V +1 CR +1 06/29/23, 09:27

  13. Benni Mack updated the commit message in patch set 8 CR +1 V +1 06/30/23, 09:49

  14. Benni Mack commented on patch set 8 06/30/23, 09:49

    We need a new issue which this patch actually resolves
  15. core-ci voted on patch set 8 V +1 06/30/23, 09:57 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43144

  16. Benni Mack voted on patch set 8 V +1 CR +1 07/04/23, 20:06

  17. Benni Mack commented on patch set 8 CR -1 07/05/23, 11:49

    we still need a ticket
  18. Stefan Bürk uploaded patch set 9 CR -1 V +1 07/05/23, 15:47

  19. core-ci voted on patch set 9 V -1 07/05/23, 16:02 Core CI is not happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43550

  20. Stefan Bürk rebased onto patch set 10 V -1 07/05/23, 16:35

  21. core-ci voted on patch set 10 V +1 07/05/23, 16:42 Core CI is happy: https://git.typo3.org/typo3/CI/cms/-/pipelines/43559

  22. Stefan Bürk voted on patch set 10 V +1 CR +1 07/05/23, 17:04

  23. Benni Mack voted on patch set 10 V +1 CR +1 07/05/23, 17:06

  24. Stefan Bürk voted on patch set 10 V +2 CR +2 07/05/23, 17:18

  25. Stefan Bürk merged patch set 11 07/05/23, 17:18