robots.txt allow root only, disallow everything else?

后端 未结 2 604
清歌不尽
清歌不尽 2021-02-01 12:47

I can\'t seem to get this to work but it seems really basic.

I want the domain root to be crawled

http://www.example.com

But nothing el

2条回答
  •  太阳男子
    2021-02-01 13:42

    When you look at the google robots.txt specifications, you can see that:

    Google, Bing, Yahoo, and Ask support a limited form of "wildcards" for path values. These are:

    1. * designates 0 or more instances of any valid character
    2. $ designates the end of the URL

    see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt?hl=en#example-path-matches

    Then as eywu said, the solution is

    user-agent: *
    Allow: /$
    Disallow: /
    

提交回复
热议问题