Should I remove meta-robots (index, follow) when I have a robots.txt?

后端 未结 2 1283
长发绾君心
长发绾君心 2020-12-11 14:04

I\'m a bit confused whether I should remove the robots meta tag, if I want search engines to follow my robots.txt rules.

If the robots meta-tag (index, follow) exist

2条回答
  •  时光说笑
    2020-12-11 14:52

    If a search engine’s bot honors your robots.txt, and you disallow crawling of /foo, then the bot will never crawl pages whose URL paths start with /foo. Hence the bot will never know that there are meta-robots elements.

    Conversely, this means that if you want to disallow indexing a page (by specyfing meta-robots with noindex), you should not disallow crawling of this page in your robots.txt. Otherwise the noindex is never accessed, and the bot thinks that crawling is forbidden, not indexing.

提交回复
热议问题