#85127linkvalidator: Add possibility to exclude specific external URLs / domains or patterns
Description
Sometimes there may be problems with external sites that linkvalidator / Guzzle doesn’t handle correctly or which can’t be handled correctly at all.
To make linkvalidator still usable, it would be good to be able to exclude certain domains / urls using regular expressions. This should be supplied in a way that an admin user can edit it.
Prerequisite: Provide general configuration for extension. This does not need to be made configurable per page tree.
Examples
URLs that work in browser, but check will fail:
Reviews 0
No change resolves or names this issue in its commit message, and no comment links one.
Related issues 5
- Bug #86918Linkvalidator stops working on specific links (external URLs)
- Feature #89457Add possibility to mark as error specific external URLs / domains or patterns
- Feature #96447Linkvalidator concept & open decisions
- Bug #99909False positive broken links by parsing URLs not inside <a> tags
- Bug #101670Linkvalidator reports some external URLs as "false positives"
History 14
-
Sybille Peters commented and changed subject
Add configuration option: file with regex patterns to exclude for external link checkingAdd possibility to exclude specific external URLs / domains or patterns <span data-processor="relativetime" data-value="2019-09-26T11:04:49+00:00" > 09/26/19, 11:04 </span> -
Sybille Peters changed Related to #86918 <span data-processor="relativetime" data-value="2019-09-26T11:05:03+00:00" > 09/26/19, 11:05 </span>
-
Lina Wolf changed Related to #89457 <span data-processor="relativetime" data-value="2019-10-18T20:36:49+00:00" > 10/18/19, 20:36 </span>
-
Sybille Peters changed subject
Add possibility to exclude specific external URLs / domains or patternslinkvalidator: Add possibility to exclude specific external URLs / domains or patterns <span data-processor="relativetime" data-value="2020-03-24T21:02:47+00:00" > 03/24/20, 21:02 </span> -
Sybille Peters changed description <span data-processor="relativetime" data-value="2020-09-07T17:02:21+00:00" > 09/07/20, 17:02 </span>
-
Sybille Peters changed removed the assignee <span data-processor="relativetime" data-value="2021-04-19T13:36:46+00:00" > 04/19/21, 13:36 </span>
-
Christopher Schnell commented <span data-processor="relativetime" data-value="2023-04-26T09:04:04+00:00" > 04/26/23, 09:04 </span>
Since this feature request is still open after almnost five years, it seems that there is not much interest in implementing this feature.
However, another common use case, where the described feature could be very useful, is for internal environments. Imagine an intranet in combination with a separated DMS application, a bugtracker or something else, where a lot of links from the TYPO3 intranet point to the other systems and which the linkvalidator alerts as false positive, because it has no access to it, maybe because of SSO.
Instead of getting a lot of false positives, the domains of these external systems could be excluded from the link checking. -
Sybille Peters commented <span data-processor="relativetime" data-value="2023-04-26T10:28:43+00:00" > 04/26/23, 10:28 </span>
@Christopher There is currently no simple way to do this with Linkvalidator out of the box.
However, Link Validator is very well customizable. You can create your own custom Link type, extending the external link type. Please see the Link Validator documentation” and look in the source code of ExternalLinktype.php
In fact, I would personally recommend to do that and not use the ExternalLinktype without customization anyway, for the following reasons:
- problem of false positives: Links are detected as broken even if they are not. This seems to be a problem which is getting worse. There are a number of reasons for this but the most common seem to be (1) incomplete certificate chain of the server (which is resolved in the browser because the browser fetches intermediate certificates, but curl (which is by default used by Guzzle, which is used by TYPO3) does not and (2) Cloudflare protected sites. If you customize the class, you can define what is checked and what is not and if specific error types or domains should be excluded
- if there are lots of external links, it might make sense to implement a crawl delay and make sure external sites are not checked too often
Alternatively, you can look at my extension Broken Link Fixer (brofix), which is based on linkvalidator but has implemented some changes, such as:
- link target cache and crawl delay when fetching to reduce access of external sites
- Button in link list for “ignoring” a target URL by domain or URL (exclude list), exclude list can be maintained in the BE moduleYou can look at the implementation in “Broken Link Fixer” (brofix) as a start to create your own or you can try out the extension.
Even with the “ignore” button, I have now deactivated the external link checking for our university TYPO3 installation except for a few known internal domains in the university because the “false positives” was too much of a problem.
-
Christopher Schnell commented <span data-processor="relativetime" data-value="2023-04-26T12:23:42+00:00" > 04/26/23, 12:23 </span>
@speters wow, just read your brofix extension documentation and this is exactly what I have been having in mind (also having an ignore button in the list). Thank you very much for the hint and your work on that extension.
Can’t this get into the core, replacing the Linkvalidator?
-
Sybille Peters commented <span data-processor="relativetime" data-value="2023-04-26T12:48:45+00:00" > 04/26/23, 12:48 </span>
@Christopher I can’t make that decision.
I am just a contributor, I am not in core team.
You can contact me on Slack if you have further questions. Anyway, the “ignore button” is just a workaround, it is not a silver bullet and the problem of the “false positives” is quite tricky to solve.
If you want to talk about linkvalidator you can also use the core channel, see contribution guide: https://docs.typo3.org/m/typo3/guide-contributionworkflow/main/en-us/Community/Index.html
-
Sybille Peters changed Related to #99909 <span data-processor="relativetime" data-value="2023-08-13T06:31:17+00:00" > 08/13/23, 06:31 </span>
-
Georg Ringer changed Related to #101670 <span data-processor="relativetime" data-value="2026-10-06T20:32:12+00:00" > 10/06/26, 20:32 </span>
-
Georg Ringer commented and changed status
NewClosed <span data-processor="relativetime" data-value="2026-10-23T06:57:37+00:00" > 10/23/26, 06:57 </span>closing the issue to cleanup forge a bit. I set a relation to the main issue #96447
-
Georg Ringer changed Related to #96447 <span data-processor="relativetime" data-value="2026-10-23T06:57:51+00:00" > 10/23/26, 06:57 </span>
Problem
In some rare cases, the checking of external URLs fails via linkvalidator even if they are not broken. See also #86918. This leads to false negatives (URLs reported as broken which are not).
One of the most annoying things for editors when working through the list of broken links are false negatives which keep coming up, clutter up the list, cannot be removed and make it really tedious to actually work through the list and fix broken links.
Proposed Solution
It would be ideal to optimize the crawl process so it always correctly reports broken links but this may not be entirely possible.
So as alternative we could add a mechanism to exclude specific URLs or URL patterns, e.g.
This could be done by just adding optional files or could be done in the GUI with an “ignore” button.