I\'m a bit confused whether I should remove the robots meta tag, if I want search engines to follow my robots.txt rules.
If the robots meta-tag (index, follow) exist
If a search engine’s bot honors your robots.txt, and you disallow crawling of /foo
, then the bot will never crawl pages whose URL paths start with /foo
. Hence the bot will never know that there are meta
-robots
elements.
Conversely, this means that if you want to disallow indexing a page (by specyfing meta
-robots
with noindex
), you should not disallow crawling of this page in your robots.txt. Otherwise the noindex
is never accessed, and the bot thinks that crawling is forbidden, not indexing.