Issue Task Under Review

#105657Enforce Strict cHash validation

Torben Hansen 11/20/24, 16:20 updated 11/09/26, 14:56

Description

Reviews 1

History 914

  1. Torben Hansen commented and changed subject Breaking - Enforce Strict cHash validation Enforce Strict cHash validation <span data-processor="relativetime" data-value="2024-11-23T17:35:15+00:00" > 11/23/24, 17:35 </span>

    Breaking change for v14

  2. Benni Mack commented <span data-processor="relativetime" data-value="2024-12-10T17:33:17+00:00" > 12/10/24, 17:33 </span>

    I do recall that some people might have issues with this setting…

  3. Sybille Peters commented <span data-processor="relativetime" data-value="2026-05-05T08:36:55+00:00" > 05/05/26, 08:36 </span>

    This is just a fyi that Semrush bot removes the cHash from the URLs it crawls as subsitemaps. While this is just one bot, other bots may do this, it might lead to potential problems (sitemaps not being loaded) when you enforce always cHash (and it’s not something you can really control from my understanding).

    Ideally, the subsitemaps should not have a cHash in URLs (e.g. use routes without query parameters like in other cases).

    *.*.*.* - - [29/Apr/2026:11:47:09 +0200] "GET /sitemap.xml?sitemap=zsb HTTP/1.1" 200 501 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" mydomain [lid -] TLSv1.3 0s 48459us
    *.*.*.* - - [29/Apr/2026:11:47:11 +0200] "GET /sitemap.xml?sitemap=news HTTP/1.1" 200 501 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" mydomain [lid -] TLSv1.3 0s 85383us
    *.*.*.* - - [29/Apr/2026:11:47:14 +0200] "GET /sitemap.xml?sitemap=unioljobs HTTP/1.1" 200 501 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" mydomain [lid -] TLSv1.3 0s 48571us
    *.*.*.* - - [29/Apr/2026:11:47:16 +0200] "GET /sitemap.xml?sitemap=branchUni HTTP/1.1" 200 501 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" mydomain [lid -] TLSv1.3 0s 50934us
    *.*.*.* - - [29/Apr/2026:11:47:17 +0200] "GET /sitemap.xml?page=1&sitemap=branchUni HTTP/1.1" 200 501 "-" "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" mydomain [lid -] TLSv1.3 0s 52329us

    The bot is loading the subsitemaps without the cHash.

    These URLs are pretty new because I recently split up the main sitemap into several branch sitemaps (all containing subsitemaps), so URLs like /sitemap_fak5.xml?page=4&sitemap=fak5… wouldn’t have existed before 2026. Also, all my subsitemaps have cHashes in the main sitemap, so it really looks like the bot is just chopping them off (for whatever reason).

    My sitemaps typically look like this:

    /sitemap.xml:

    • /sitemap.xml?sitemap=zsb&cHash=4ffc00cd972c228dd3cd9d267ccb3ca4
    • /sitemap.xml?sitemap=news&cHash=b1a8fc3b11870d1e81485bc434c2412b
    • etc.