Review main Merged

6990[BUGFIX] Links on external pages don't get indexed

Mario Rimann CR +2 V +2 Normal +128 −3 updated 07/25/13, 08:14

Commit message

Allows the crawler to start indexing a specific file like
www.domain.tld/foobar.html instead of just www.domain.tld/

This is just about the comparison against the base URL and
enables the Crawler to start crawling at e.g. a file that contains
a manually generated list of links to follow. Before that change,
even links to targets on the same domain were rejected by
the checkUrl() method in case the base Url was pointing to some
file instead of "/". This was because the base URL was then not
part of the target URL.
After stripping off any path from the base URL for this comparison
this can now also be used to start crawling from a file.

Review

  • Code review CR +2

    Imported before Forger kept the single votes, the next import adds them.

  • Verified V +2

    Imported before Forger kept the single votes, the next import adds them.

Issues 1

Files 2