Will this robots.txt only allow googlebot to index my site?

后端 未结 3 1672

Will this robots.txt file only allow googlebot to index my site\'s index.php file? CAVEAT, I have an htaccess redirect that people who type in

http://www.example

3条回答
  •  闹比i
    闹比i (楼主)
    2021-01-27 13:36

    Not really.

    Good bots
    Only "good" bots follow the robots.txt instructions (not all robots and spiders bother to read/follow robots.txt). That might not even include all the main search engine's bots, but it definitely mean that some web crawlers will just completely ignore your requests (you should look at using .htaccess or password protection if you really want to stop bots/crawlers from seeing parts of your site).

    Second checks
    Google makes multiple visits to your website, including appearing as a browsing user. This second visit will ignore the robots.txt file. The second visit probably doesn't actually index (if that's your worry) but it does check to make sure you're not trying to fool the indexing bot (for SEO etc).

    That being said your syntax is right... if that's all you're asking, then yes it'll work, just not as well as you might hope.

提交回复
热议问题