Issue Bug Closed

#105753Linkvalidator does not parse domains with Umlauts correctly

Roman Schilter 12/03/24, 14:34 updated 12/02/25, 14:31

Description

Steps to reproduce:

It will find the link “https://geb” and report it as broken.

The reason is the Regexp in UrlSoftReferenceParser does not match Umlauts.

Reviews 3

Files 1

History 813

  1. Sybille Peters commented and changed attached linkvalidator_result.png status New Accepted <span data-processor="relativetime" data-value="2024-12-30T10:47:02+00:00" > 12/30/24, 10:47 </span>

    I can reproduce this in v14 (main) and v12 (latest 12.4 branch using page type “link to external page” and the link in pages.url (as described).

    I cannot reproduce it with content element CType=“textmedia” and links in tt_content.bodytext (RTE). There, the links are parsed correctly.

  2. Sybille Peters commented <span data-processor="relativetime" data-value="2024-12-30T10:52:35+00:00" > 12/30/24, 10:52 </span>

    The following works correctly in RTE:

    select bodytext from tt_content where pid=913;
    <p>mit Umlauten <a href="https://xn--gebudehlle-s5a60a.swiss/">link (umlauts)</a></p>
    <p><a href="https://xn--gebudehlle-s5a60a.swiss/sdfsddfs">broken link (umlauts)</a></p>
    <p>Encoded: <a href="https://gebäudehülle.swiss">link (encoded)</a></p>
    <p><a href="https://gebäudehülle.swiss/sdfsddfs">broken link (encoded)</a></p>
  3. Sybille Peters commented <span data-processor="relativetime" data-value="2024-12-30T11:09:48+00:00" > 12/30/24, 11:09 </span>

    For pages.url, if the already encoded value is used, it is ok:

    Not ok, if this is used:

    As a workaround you can use the encoded URL (the URL is already converted for me, when I copy it from the browser address bar, for example).

    As mentioned above, the problem is in UrlSoftReferenceParser. Possible solutions:

    • value is already converted when saved in pages.url (and other fields)
    • linkvalidator does not use softref parser to “parse” fields where the content can be used as is (e.g. for pages.url which is of type “input” with “softref” = “url”)
    • linkvalidator converts the URL before passing it to the sofref parser (not possible for fields where parsing should be performed FIRST)
    • the softref parser UrlSoftReferenceParser converts the URL first. We already have code for this in ExternalLinktype::preprocessUrl :
    protected function preprocessUrl(string $url): string
        {
            $url = html_entity_decode($url);
            $parts = parse_url($url);
            if ($parts['host'] ?? false) {
                try {
                    $newDomain = (string)idn_to_ascii($parts['host']);
                    if (strcmp($parts['host'], $newDomain) !== 0) {
                        $parts['host'] = $newDomain;
                        $url = HttpUtility::buildUrl($parts);
                    }
                } catch (\Exception | \Throwable $e) {
                    // ignore error and proceed with link checking
                }
            }
            return $url;
        }
  4. Sybille Peters commented <span data-processor="relativetime" data-value="2024-12-30T11:32:08+00:00" > 12/30/24, 11:32 </span>

    Also, there is no problem with parsing sys_file_reference.link because it uses the TypolinkSoftReferenceParser.

    I am wondering what is the point of having UrlSoftReferenceParser because TypolinkSoftReferenceParser can also parse urls.

    Also, pages.url could use type=link and with allowedTypes=‘url’ (though that would not solve the problem with possibly other fields)

    pages.url:
    - type=input
    - softref=url

    sys_file_reference.link:
    - type=link
    - softref=typolink

  5. Gerrit Code Review pushed patch set 1 for main to change 87609 status Accepted Under Review <span data-processor="relativetime" data-value="2025-01-01T09:57:38+00:00" > 01/01/25, 09:57 </span>

  6. Sybille Peters commented and changed status Under Review Resolved done 0 % 100 % <span data-processor="relativetime" data-value="2025-01-02T10:45:08+00:00" > 01/02/25, 10:45 </span>

    Applied in changeset commit:22265db3b933a7d2e40625a1f65e0eec73cbc2dd.