How can I tell Google not to crawl a set of Urls

问题

How do I stop google to crawl to certain urls in my application?

For example: I want google to stop crawling all the URLs that starts with http://www.myhost-test.com/

What should I add in my robot.txt?

回答1:

The answer can be found directly here:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

But it looks like you add "disallow" and your url.

来源：https://stackoverflow.com/questions/11542918/how-can-i-tell-google-not-to-crawl-a-set-of-urls