Review main Merged

6990[BUGFIX] Links on external pages don't get indexed

Mario Rimann CR +2 V +2 Normal +128 −3 updated 07/25/13, 08:14

Commit message

Allows the crawler to start indexing a specific file like
www.domain.tld/foobar.html instead of just www.domain.tld/

This is just about the comparison against the base URL and
enables the Crawler to start crawling at e.g. a file that contains
a manually generated list of links to follow. Before that change,
even links to targets on the same domain were rejected by
the checkUrl() method in case the base Url was pointing to some
file instead of "/". This was because the base URL was then not
part of the target URL.
After stripping off any path from the base URL for this comparison
this can now also be used to start crawling from a file.

Review

  • Code review CR +2
    • Georg Ringer +2
    • Michael Stucki +1
  • Verified V +2
    • Georg Ringer +2
    • Michael Stucki +1

Issues 1

Files 2

Review log 48

Show 23 earlier messages
  1. Mario Rimann commented on patch set 1 05/27/13, 17:19

    (1 inline comment)
  2. Mario Rimann uploaded patch set 2 05/27/13, 17:19

  3. Mario Rimann commented on patch set 2 05/27/13, 17:19

    To what shall the Releases-Line get changed in your opinion?
  4. Mario Rimann uploaded patch set 3 05/27/13, 17:19

  5. Mario Rimann commented on patch set 3 05/27/13, 17:19

    Ok, seems like a rebase is needed to be on top of all the CGL-fixing commits of the past months...
  6. Mario Rimann uploaded patch set 4 05/27/13, 17:19

  7. Mario Rimann commented on patch set 5 05/27/13, 17:19

    @Dmitry: I think you've got that wrong. The change affects only the method that checks if a given URL is allowed to be indexed - the whole indexing of the (unmodified) url is untouched! Can you please re-check?
  8. Mario Rimann uploaded patch set 5 05/27/13, 17:19

  9. Mario Rimann commented on patch set 5 05/27/13, 17:19

    Patchset 5 is a manual rebase of the change. Would be nice if this could be finally decided now - either accept it, or decline it.
  10. Dmitry Dulepov commented on patch set 5 05/27/13, 17:19

    No score Cannot review now but removed a -1.
  11. Dmitry Dulepov commented on patch set 5 05/27/13, 17:19

    I would prefer that you didn't submit this This does not solve the issue. Imagine that you want to index http://example.com/subdir1/ and not http://example.com/subdir2/. Your patch will index the whole http://example.com/ instead of the content of just /subdir1/. I think the existing behavior is on purpose and will be broken with this patch. Sorry, Mario :(
  12. Stefan Neufeind commented on patch set 1 1 comment 05/27/13, 17:19

    I would prefer that you didn't submit this
  13. Stefan Neufeind commented on patch set 2 05/27/13, 17:19

    Looks good to me, but someone else must approve
  14. Stefan Neufeind commented on patch set 2 05/27/13, 17:19

    adjust Releases-line before commit
  15. Stefan Neufeind commented on patch set 4 05/27/13, 17:19

    Looks good to me, but someone else must approve seems to make sense by reading
  16. Stefan Neufeind commented on patch set 5 05/27/13, 17:19

    Dmitry, please recheck and/or followup with Mario.
  17. Philipp Gampe commented on patch set 2 05/27/13, 17:19

    (1 inline comment)
  18. Simon Schaufelberger commented on patch set 1 05/27/13, 17:19

    (2 inline comments)

    class.crawler.php:574

    ja wo geht just denn hin?

    class.crawler.php:576

    is there no api for that? this looks quite hard coded...
  19. Simon Schaufelberger commented on patch set 2 05/27/13, 17:19

    Looks good to me, but someone else must approve
  20. Mr. Jenkins commented on patch set 1 05/27/13, 17:19

  21. Mr. Jenkins commented on patch set 2 05/27/13, 17:19

  22. Mr. Jenkins commented on patch set 3 05/27/13, 17:19

    Fails; I would prefer that you didn't submit this Build Failed https://ci.typo3.org/job/typo3-v4-core-gerrit/4626/ : FAILURE
  23. Mr. Jenkins commented on patch set 4 05/27/13, 17:19

  1. Mr. Jenkins commented on patch set 5 05/27/13, 17:19

  2. Wouter Wolters commented on patch set 4 05/27/13, 17:19

    Looks good to me, but someone else must approve Can't test this, but looks oké for me.
  3. Mario Rimann uploaded patch set 6 07/11/13, 11:03

  4. Mario Rimann commented on patch set 6 07/11/13, 11:04

    Once again rebased - would be nice if we could get this into the next release :-)
  5. Mr. Jenkins commented on patch set 6 07/11/13, 11:04

  6. Stefan Neufeind commented on patch set 6 CR -1 07/11/13, 11:31

    This would break functionality for limiting crawling to a certain subdirectory like http://example.com/~user1/ So instead of fixing up the baseUrl-value here, imho it should be taken care of where the value comes from - maybe even allow giving a separate baseURL (which really is the base then) and specifying a start-URL.

    class.crawler.php:576 Reply

    haven't found one unfortunately. But searching for it I saw that the port-number (if given) might also be useful. t3lib/class.t3lib_htmlmail.php: return $addr['scheme'] . '://' . $addr['host'] . ($addr['port'] ? ':' . $addr['port'] : '') . $ref;
  7. Mario Rimann commented on patch set 6 07/14/13, 19:56

    @Stefan: I think you've got the patch wrong. The change only affects the method that acts as a gate-keeper by verifying that an actual URL to be indexed is part of a given base-URL. The indexing itself happens in another place of indexed_search. So what this does: URL to index: http://www.foo.tld/bar Base-URL: http://www.foo.tld/ -> that's valid (and was before my change, too) URL to index: http://www.foo.tld/bar Base-URL (e.g. a list of URLs to fetch + follow): http://www.foo.tld/listOfVeryImportantStuff.html -> would not be accepted before the change -> will be allowed after the change And IMHO the case you've mentioned should be possible, too (indexing user-subdirectories). Or to name it short: Before the change, the starting point of an external indexing job needed to be the root of e.g. a domain - now it can also be a separate file on a domain that contains links. @Stefan: Could you please re-check?

    class.crawler.php:574 Reply

    haha, took me a moment to get the joke :-) Will upload a new patch set :-)
  8. Christian Kuhn commented on patch set 6 07/14/13, 21:27

    Maybe just add some unit tests expressing the different input values and its answers to unconfuse the confusion ;) ?
  9. Mario Rimann uploaded patch set 7 07/15/13, 21:33

  10. Mr. Jenkins commented on patch set 7 07/15/13, 21:34

  11. Georg Ringer voted on patch set 7 V +1 CR +1 07/16/13, 05:57

  12. Pascal Maechler voted on patch set 7 CR +1 07/16/13, 06:59

  13. Wouter Wolters commented on patch set 7 CR -1 07/16/13, 09:49

    Code looks ok in general, one small nitpick. I will provide an update this evening if not taken care at that time.

    CrawlerHookTest.php:19

    one tab less
  14. Mario Rimann uploaded patch set 8 07/16/13, 10:03

  15. Mr. Jenkins commented on patch set 8 07/16/13, 10:03

  16. Mario Rimann commented on patch set 8 07/16/13, 10:03

    Thanks Wouter, Patch-Set 8 changes the indention of the class comment block.
  17. Wouter Wolters commented on patch set 8 CR +1 07/16/13, 10:36

    Testing will follow this evening
  18. Michael Stucki uploaded patch set 9 07/16/13, 10:38

  19. Mr. Jenkins commented on patch set 9 07/16/13, 10:39

  20. Michael Stucki commented on patch set 9 V +1 CR +1 07/16/13, 10:41

    Call me a nitpicker, but I would like to see more familiar example URLs... ;-) Giving +1 since I just changed the unit test. After all, this is almost a no-brainer!
  21. Georg Ringer voted on patch set 9 V +2 CR +2 07/16/13, 11:44

  22. Georg Ringer merged patch set 10 07/16/13, 11:44

  23. Jigal van Hemert commented on patch set 10 07/16/13, 13:37

    Fails: Add these unit tests /** * @test */ public function checkUrlReturnsFalseOnUrlOutsideTheBaseUrl() { $this->assertFalse( $this->fixture->checkUrl( 'http://typo3.org/about.html';, array(), 'http://typo3.org/extensions/'; ) ); } /** * @test */ public function checkUrlReturnsFalseOnUrlOutsideTheBaseUrlWithPath() { $this->assertFalse( $this->fixture->checkUrl( 'http://typo3.org/about.html';, array(), 'http://typo3.org/extensions/index.html'; ) ); } The function clearly states in the doc header: "input URL must be "inside" the base URL!" Stefan and me discussed this briefly when he gave his -1 and had to come to the conclusion that it is impossible to know whether a URL points to a file or a directory if it doesn't end in a slash. The only solution would be to set the base url and the starting url independently and check if the starting url is "inside" the base URL. This means that it's a feature and could only go into master.
  24. Mario Rimann commented on patch set 10 07/16/13, 20:46

    @Jigal: I think you're right, that the comment of that method is now not 100% matching the decision that method executes. But so far, I wasn't aware of that use-case. So far, I knew this functionality only to crawl e.g. a foreign domain's content - where baseURL and start-URL are equal. If you're seeing/needing that use-case to limit indexing to a specific subdirectory (e.g. baseURL = http://foo.tld/indexing/), we should rethink the whole thing and I agree this should get a follow up then (as a feature with split up baseURL and start-URL probably). (Sorry Stefan, I now re-read your comment - but I only got your idea after Jigals comment + examples) If you're just referring to the comment of that method to be wrong now, I wouldn't care too much: The comment of that method was partially wrong since TYPO3 4.0 and no one cared. Please contact me off-review-and-off-bugtracker to discuss the issue and initiate the next step if needed.
  25. Jigal van Hemert commented on patch set 10 07/25/13, 08:14

    Reverted This patchset was reverted in change: Iebb62c89959328cc460fea5f1f46fb824b892822