6990[BUGFIX] Links on external pages don't get indexed
Commit message
Review
-
Code review CR +2
- Georg Ringer +2
- Michael Stucki +1
-
Verified V +2
- Georg Ringer +2
- Michael Stucki +1
Issues 1
Files 2
Review log 48
Show 23 earlier messages
-
Mario Rimann commented on patch set 1 05/27/13, 17:19
(1 inline comment) -
Mario Rimann uploaded patch set 2 05/27/13, 17:19
-
Mario Rimann commented on patch set 2 05/27/13, 17:19
To what shall the Releases-Line get changed in your opinion? -
Mario Rimann uploaded patch set 3 05/27/13, 17:19
-
Mario Rimann commented on patch set 3 05/27/13, 17:19
Ok, seems like a rebase is needed to be on top of all the CGL-fixing commits of the past months... -
Mario Rimann uploaded patch set 4 05/27/13, 17:19
-
Mario Rimann commented on patch set 5 05/27/13, 17:19
@Dmitry: I think you've got that wrong. The change affects only the method that checks if a given URL is allowed to be indexed - the whole indexing of the (unmodified) url is untouched! Can you please re-check? -
Mario Rimann uploaded patch set 5 05/27/13, 17:19
-
Mario Rimann commented on patch set 5 05/27/13, 17:19
Patchset 5 is a manual rebase of the change. Would be nice if this could be finally decided now - either accept it, or decline it. -
Dmitry Dulepov commented on patch set 5 05/27/13, 17:19
No score Cannot review now but removed a -1. -
Dmitry Dulepov commented on patch set 5 05/27/13, 17:19
I would prefer that you didn't submit this This does not solve the issue. Imagine that you want to index http://example.com/subdir1/ and not http://example.com/subdir2/. Your patch will index the whole http://example.com/ instead of the content of just /subdir1/. I think the existing behavior is on purpose and will be broken with this patch. Sorry, Mario :( -
Stefan Neufeind commented on patch set 1 1 comment 05/27/13, 17:19
I would prefer that you didn't submit this -
Stefan Neufeind commented on patch set 2 05/27/13, 17:19
Looks good to me, but someone else must approve -
Stefan Neufeind commented on patch set 2 05/27/13, 17:19
adjust Releases-line before commit -
Stefan Neufeind commented on patch set 4 05/27/13, 17:19
Looks good to me, but someone else must approve seems to make sense by reading -
Stefan Neufeind commented on patch set 5 05/27/13, 17:19
Dmitry, please recheck and/or followup with Mario. -
Philipp Gampe commented on patch set 2 05/27/13, 17:19
(1 inline comment) -
Simon Schaufelberger commented on patch set 1 05/27/13, 17:19
(2 inline comments)class.crawler.php:574ja wo geht just denn hin?class.crawler.php:576is there no api for that? this looks quite hard coded... -
Simon Schaufelberger commented on patch set 2 05/27/13, 17:19
Looks good to me, but someone else must approve -
Mr. Jenkins commented on patch set 1 05/27/13, 17:19
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/2554/ : SUCCESS -
Mr. Jenkins commented on patch set 2 05/27/13, 17:19
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/2568/ : SUCCESS -
Mr. Jenkins commented on patch set 3 05/27/13, 17:19
Fails; I would prefer that you didn't submit this Build Failed https://ci.typo3.org/job/typo3-v4-core-gerrit/4626/ : FAILURE -
Mr. Jenkins commented on patch set 4 05/27/13, 17:19
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/4628/ : SUCCESS
-
Mr. Jenkins commented on patch set 5 05/27/13, 17:19
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/8180/ : SUCCESS -
Wouter Wolters commented on patch set 4 05/27/13, 17:19
Looks good to me, but someone else must approve Can't test this, but looks oké for me. -
Mario Rimann uploaded patch set 6 07/11/13, 11:03
-
Mario Rimann commented on patch set 6 07/11/13, 11:04
Once again rebased - would be nice if we could get this into the next release :-) -
Mr. Jenkins commented on patch set 6 07/11/13, 11:04
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/9565/ : SUCCESS -
Stefan Neufeind commented on patch set 6 CR -1 07/11/13, 11:31
This would break functionality for limiting crawling to a certain subdirectory like http://example.com/~user1/ So instead of fixing up the baseUrl-value here, imho it should be taken care of where the value comes from - maybe even allow giving a separate baseURL (which really is the base then) and specifying a start-URL.class.crawler.php:576Replyhaven't found one unfortunately. But searching for it I saw that the port-number (if given) might also be useful. t3lib/class.t3lib_htmlmail.php: return $addr['scheme'] . '://' . $addr['host'] . ($addr['port'] ? ':' . $addr['port'] : '') . $ref; -
Mario Rimann commented on patch set 6 07/14/13, 19:56
@Stefan: I think you've got the patch wrong. The change only affects the method that acts as a gate-keeper by verifying that an actual URL to be indexed is part of a given base-URL. The indexing itself happens in another place of indexed_search. So what this does: URL to index: http://www.foo.tld/bar Base-URL: http://www.foo.tld/ -> that's valid (and was before my change, too) URL to index: http://www.foo.tld/bar Base-URL (e.g. a list of URLs to fetch + follow): http://www.foo.tld/listOfVeryImportantStuff.html -> would not be accepted before the change -> will be allowed after the change And IMHO the case you've mentioned should be possible, too (indexing user-subdirectories). Or to name it short: Before the change, the starting point of an external indexing job needed to be the root of e.g. a domain - now it can also be a separate file on a domain that contains links. @Stefan: Could you please re-check?class.crawler.php:574Replyhaha, took me a moment to get the joke :-) Will upload a new patch set :-) -
Christian Kuhn commented on patch set 6 07/14/13, 21:27
Maybe just add some unit tests expressing the different input values and its answers to unconfuse the confusion ;) ? -
Mario Rimann uploaded patch set 7 07/15/13, 21:33
-
Mr. Jenkins commented on patch set 7 07/15/13, 21:34
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/9622/ : SUCCESS -
Georg Ringer voted on patch set 7 V +1 CR +1 07/16/13, 05:57
-
Pascal Maechler voted on patch set 7 CR +1 07/16/13, 06:59
-
Wouter Wolters commented on patch set 7 CR -1 07/16/13, 09:49
Code looks ok in general, one small nitpick. I will provide an update this evening if not taken care at that time.CrawlerHookTest.php:19one tab less -
Mario Rimann uploaded patch set 8 07/16/13, 10:03
-
Mr. Jenkins commented on patch set 8 07/16/13, 10:03
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/9627/ : SUCCESS -
Mario Rimann commented on patch set 8 07/16/13, 10:03
Thanks Wouter, Patch-Set 8 changes the indention of the class comment block. -
Wouter Wolters commented on patch set 8 CR +1 07/16/13, 10:36
Testing will follow this evening -
Michael Stucki uploaded patch set 9 07/16/13, 10:38
-
Mr. Jenkins commented on patch set 9 07/16/13, 10:39
Build Successful https://ci.typo3.org/job/typo3-v4-core-gerrit/9628/ : SUCCESS -
Michael Stucki commented on patch set 9 V +1 CR +1 07/16/13, 10:41
Call me a nitpicker, but I would like to see more familiar example URLs... ;-) Giving +1 since I just changed the unit test. After all, this is almost a no-brainer! -
Georg Ringer voted on patch set 9 V +2 CR +2 07/16/13, 11:44
-
Georg Ringer merged patch set 10 07/16/13, 11:44
-
Jigal van Hemert commented on patch set 10 07/16/13, 13:37
Fails: Add these unit tests /** * @test */ public function checkUrlReturnsFalseOnUrlOutsideTheBaseUrl() { $this->assertFalse( $this->fixture->checkUrl( 'http://typo3.org/about.html';, array(), 'http://typo3.org/extensions/'; ) ); } /** * @test */ public function checkUrlReturnsFalseOnUrlOutsideTheBaseUrlWithPath() { $this->assertFalse( $this->fixture->checkUrl( 'http://typo3.org/about.html';, array(), 'http://typo3.org/extensions/index.html'; ) ); } The function clearly states in the doc header: "input URL must be "inside" the base URL!" Stefan and me discussed this briefly when he gave his -1 and had to come to the conclusion that it is impossible to know whether a URL points to a file or a directory if it doesn't end in a slash. The only solution would be to set the base url and the starting url independently and check if the starting url is "inside" the base URL. This means that it's a feature and could only go into master. -
Mario Rimann commented on patch set 10 07/16/13, 20:46
@Jigal: I think you're right, that the comment of that method is now not 100% matching the decision that method executes. But so far, I wasn't aware of that use-case. So far, I knew this functionality only to crawl e.g. a foreign domain's content - where baseURL and start-URL are equal. If you're seeing/needing that use-case to limit indexing to a specific subdirectory (e.g. baseURL = http://foo.tld/indexing/), we should rethink the whole thing and I agree this should get a follow up then (as a feature with split up baseURL and start-URL probably). (Sorry Stefan, I now re-read your comment - but I only got your idea after Jigals comment + examples) If you're just referring to the comment of that method to be wrong now, I wouldn't care too much: The comment of that method was partially wrong since TYPO3 4.0 and no one cared. Please contact me off-review-and-off-bugtracker to discuss the issue and initiate the next step if needed. -
Jigal van Hemert commented on patch set 10 07/25/13, 08:14
Reverted This patchset was reverted in change: Iebb62c89959328cc460fea5f1f46fb824b892822