Issue Feature Closed

#85679Add robot.txt functionality

Richard Haeser 07/29/18, 22:02 updated 09/24/19, 13:36

Description

TYPO3 should deliver an out-of-the-box robots.txt to prevent certain directories (like eg /typo3) to be indexed. Besides that, it should be possible by integrators to block certain parts of the website to be crawled to prevent using crawl budget while it is not needed to be in the index.

Reviews 0

No change resolves or names this issue in its commit message, and no comment links one.