Issue Bug Closed

#89617XML sitemaps should not be indexed by search engines

Chris Müller 11/08/19, 14:39 updated 02/15/23, 09:53

Description

The content of a XML sitemap should not be listed on search engine result
pages. This is confirmed by John Mueller (Webmaster Trends Analyst from Google):

https://twitter.com/JohnMu/status/1192434849216434177

Reviews 2

History 610

  1. Gerrit Code Review pushed patch set 1 for master to change 62253 status New Under Review 11/08/19, 14:42

  2. Gerrit Code Review pushed patch set 2 for master to change 62253 11/08/19, 14:54

  3. Gerrit Code Review pushed patch set 1 for 9.5 to change 62219 11/08/19, 23:42

  4. Chris Müller commented and changed status Under Review Resolved done 0 % 100 % 11/09/19, 00:00

    Applied in changeset commit:cdab79385c12d4d4bf4059f184243a93c1d4caf3.

  5. Gerrit Code Review pushed patch set 2 for 9.5 to change 62219 status Resolved Under Review 11/09/19, 07:47

  6. Chris Müller commented and changed status Under Review Resolved 11/09/19, 08:00

    Applied in changeset commit:ff6c916fbd63e90d714a89ea58f99ea567ced14a.

  7. Benni Mack changed status Resolved Closed 12/23/19, 22:44

  8. Mordamir commented 07/08/22, 16:04

    Whats the reason behind this decision?

    With the x-robots-tag set to noindex, google search console does not accept the sitemap. And a xml sitemap is mostly for crawlers. So why should i exclude the crawlers from using the sitemap?

    Maybe this is recent change from google?

  9. Chris Müller commented 07/19/22, 18:18

    “With the x-robots-tag set to noindex, google search console does not accept the sitemap.”

    I can’t reproduce that. Added some sitemaps to GSC which is possible some time ago with this change.

    You have to distinguish between crawling and indexing. Crawling means, that the crawler is able and allowed to retrieve the content and follow the links. This is possible for XML sitemaps with the “X-Robots-Tag: noindex” header. This is the same for a meta tag “robots” with “noindex” on a page: The page is crawled (and links followed) but the page isn’t indexed (= displayed in the search results).

    But: The XML sitemap should not be visible in the results of a search engine. Thus the “X-Robots-Tag: noindex”. See also the link in the description of this issue.

    And: If you want to allow the indexing of the XML sitemap (which means displaying in the search results) for whatever reason, you can easily achieve this by adding the following line in the TypoScript setup of your site package:

    seo_sitemap.config.additionalHeaders.20 >

  10. Florian Seirer commented 02/15/23, 09:53

    Had the same issue as @mordamir: Our XML sitemaps were not crawled by Google for weeks because of the noindex header, according to the report in the Google Search Console.

    But simply resubmitting the sitemaps fixed the problem. All URLs from the sitemaps are now recognized.

    Maybe this was just a temporary issue on Google’s side?