#89287Make linkvalidator crawling polite
Description
Currently, linkvalidator does not apply common practice for being “nice” / “polite” when crawling other websites:
- It should be possible to see what is crawling your site. It is usually standard to add a URL and contact information to the User-Agent or referrer, e.g.
"Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)"
"https://www.google.de/" "Mozilla/5.0 (iPhone; CPU iPhone OS 12_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1"
- the crawler should respect the robots.txt
- the crawler should wait between consecutive requests (Crawl-Delay). A good minimum value is e.g. 1-5 second for requests done on same domain
While linkvalidator is not a full web crawler which downloads the entire pages on the site and it currently uses HEAD by default (and not GET) which does not download the entire page – so this is not as dramatic.
But preferably, I think we should follow these recommendations as well.
Possible solutions
Not just for reducing load on other sites but also on the TYPO3 site, some changes might be made to the crawling process:
- when crawling external URLs, do not check right away but defer to a separate task which will handle the crawling of external URLs (with reasonable delays in between)
- do not keep crawling over and over but regularly only recrawl content which was recently modified
- optional: delegate the checking of external URLs, e.g. use a URLchecking service
This may make it necessary to make more things asynchronous and confine the link checking only to the scheduler.
Resources
About politeness of web crawlers:
URL checking site:
Reviews 1
Related issues 6
- Epic #85006Reduce falsely reported broken links
- Bug #86918Linkvalidator stops working on specific links (external URLs)
- Epic #93547Collection of problems with large sites
- Feature #96447Linkvalidator concept & open decisions
- Bug #101670Linkvalidator reports some external URLs as "false positives"
- Task #101671Disable external linktypes by default in linkvalidator
History 11
-
Sybille Peters changed tracker
BugTask <span data-processor="relativetime" data-value="2021-10-20T20:11:12+00:00" > 10/20/21, 20:11 </span> -
Sybille Peters changed Tags throttle, outgoing HTTP requests, resources, large-site <span data-processor="relativetime" data-value="2021-11-21T08:23:32+00:00" > 11/21/21, 08:23 </span>
-
Sybille Peters changed Related to #93547 <span data-processor="relativetime" data-value="2021-11-21T08:23:41+00:00" > 11/21/21, 08:23 </span>
-
Sybille Peters commented <span data-processor="relativetime" data-value="2021-11-21T08:30:22+00:00" > 11/21/21, 08:30 </span>
-
Rémy DANIEL changed Related to #85006 <span data-processor="relativetime" data-value="2023-08-06T06:47:16+00:00" > 08/06/23, 06:47 </span>
-
Sybille Peters changed Related to #101670 <span data-processor="relativetime" data-value="2023-08-13T06:53:41+00:00" > 08/13/23, 06:53 </span>
-
Sybille Peters changed Related to #101671 <span data-processor="relativetime" data-value="2023-08-13T10:47:45+00:00" > 08/13/23, 10:47 </span>
-
Garvin Hicking changed Related to #86918 <span data-processor="relativetime" data-value="2024-07-11T15:29:22+00:00" > 07/11/24, 15:29 </span>
-
Garvin Hicking commented <span data-processor="relativetime" data-value="2024-07-11T15:29:35+00:00" > 07/11/24, 15:29 </span>
(Some parts of this were addressed with https://review.typo3.org/c/Packages/TYPO3.CMS/+/61801)
-
Georg Ringer commented and changed status
NewClosed <span data-processor="relativetime" data-value="2026-10-23T06:58:45+00:00" > 10/23/26, 06:58 </span>closing the issue to cleanup forge a bit. I set a relation to the main issue #96447
-
Georg Ringer changed Related to #96447 <span data-processor="relativetime" data-value="2026-10-23T06:58:53+00:00" > 10/23/26, 06:58 </span>
Not having a throttling of outgoing
URLsHTTP requests and only caching the outgoing requests once per check cycle is one of the main reasons I do not want to use Linkvalidator in my site and I find that problematic if a TYPO3 extension bombards external sites with requests.The way I currently changed the external link checking behaviour in “brofix” (which is a fork of linkvalidator with some changes), is that I added a minimum delay per domain: If the last request to an URL of this domain is less than X seconds, we wait. This delays the checking, but since I also use a link target cache and don’t care if initial check takes several hours, I do see this as acceptable and an improvement in any case.
The wordpress plugin “Link checker” also has a throttle, see