#89617XML sitemaps should not be indexed by search engines
Description
The content of a XML sitemap should not be listed on search engine result
pages. This is confirmed by John Mueller (Webmaster Trends Analyst from Google):
Reviews 2
History 610
-
Gerrit Code Review pushed patch set 1 for master to change 62253 status
NewUnder Review 11/08/19, 14:42 -
Gerrit Code Review pushed patch set 2 for master to change 62253 11/08/19, 14:54
-
Gerrit Code Review pushed patch set 1 for 9.5 to change 62219 11/08/19, 23:42
-
Chris Müller commented and changed status
Under ReviewResolved done0 %100 % 11/09/19, 00:00 -
Gerrit Code Review pushed patch set 2 for 9.5 to change 62219 status
ResolvedUnder Review 11/09/19, 07:47 -
Chris Müller commented and changed status
Under ReviewResolved 11/09/19, 08:00Applied in changeset commit:ff6c916fbd63e90d714a89ea58f99ea567ced14a.
-
Benni Mack changed status
ResolvedClosed 12/23/19, 22:44 -
Mordamir commented 07/08/22, 16:04
Whats the reason behind this decision?
With the x-robots-tag set to noindex, google search console does not accept the sitemap. And a xml sitemap is mostly for crawlers. So why should i exclude the crawlers from using the sitemap?
Maybe this is recent change from google?
-
Chris Müller commented 07/19/22, 18:18
“With the x-robots-tag set to noindex, google search console does not accept the sitemap.”
I can’t reproduce that. Added some sitemaps to GSC which is possible some time ago with this change.
You have to distinguish between crawling and indexing. Crawling means, that the crawler is able and allowed to retrieve the content and follow the links. This is possible for XML sitemaps with the “X-Robots-Tag: noindex” header. This is the same for a meta tag “robots” with “noindex” on a page: The page is crawled (and links followed) but the page isn’t indexed (= displayed in the search results).
But: The XML sitemap should not be visible in the results of a search engine. Thus the “X-Robots-Tag: noindex”. See also the link in the description of this issue.
And: If you want to allow the indexing of the XML sitemap (which means displaying in the search results) for whatever reason, you can easily achieve this by adding the following line in the TypoScript setup of your site package:
seo_sitemap.config.additionalHeaders.20 >
-
Florian Seirer commented 02/15/23, 09:53
Had the same issue as @mordamir: Our XML sitemaps were not crawled by Google for weeks because of the noindex header, according to the report in the Google Search Console.
But simply resubmitting the sitemaps fixed the problem. All URLs from the sitemaps are now recognized.
Maybe this was just a temporary issue on Google’s side?
Applied in changeset commit:cdab79385c12d4d4bf4059f184243a93c1d4caf3.