robots.txt - is this working?

Question

I just ran into a robots.txt that looks like this:

User-agent: *
Disallow: /foobar

User-agent: badbot
Disallow: *
         


        

Answer 1

Each bot only ever complies to at most a single record (block).

A block starts with one or more User-agent lines, typically followed by Disallow lines (at least one is required). Blocks are separated by blank lines.

A bot called "badbot" will look for a record with the line User-agent: badblock (or similar, as the bot "should be liberal in interpreting this field"). If no such line is found, it will look for a record with the line User-agent: *. If even this doesn’t exist, the bot is allowed to do everything (= default).

So in your example, the bot called "badbot" will follow only the second record (you probably mean Disallow: / instead of Disallow: *), while all other bots only follow the first record.