#14798Robots.txt and indexed search
Description
I’ve got a handful of those pdf files that I don’t want
indexed. So, to satisfy google, and other search engines, I use a
robots.txt file so they aren’t indexed.
My requested feature is, getting typo to honor robots.txt files, and
skip indexing files listed in there.
(issue imported from #M1170)
Reviews 0
No change resolves or names this issue in its commit message, and no comment links one.