Robots.txt: disallow a folder's name, regardless at which depth it may show up

问题

So I have to disallow search engines from indexing our REST web service responses (it's a Sitecore website); all of them have the same name in the URL but show up at different levels in the server hierarchy, and I was wondering if I can write a "catch all" entry in our robots file or if I am doomed to write an extensive list.

Can I add something like

Disallow: */ajax/*

to catch all folders named "ajax" regardless of where they appear?

回答1:

robots.txt specification doesn't say anything about wildcards but Google (Google Robots.txt Specifications) and Bing allow the use of wildcards in robots.txt files.

Disallow: */ajax/*

Your disallow is valid for all the /ajax/ urls no matter what is the nesting level of /ajax/.

回答2:

You should be able to just use Disallow: /*ajax. Similar question over here: How to disallow service api and multilingual urls in robots.txt

来源：https://stackoverflow.com/questions/31137082/robots-txt-disallow-a-folders-name-regardless-at-which-depth-it-may-show-up