Ethics of robots.txt [closed]

Answer 1

If people make it available to public access, they shouldn't try to put limits on it. Adding a robots.txt file to your site is the equivalent to putting a sign on your lawn that says "Please don't look at me."

Answer 2

I'm showing some ignorance here, but I always thought a bot was something only sent out by a search engine. Like Google or Yahoo.

Thus, if you wrote an application that searched content on the internet, I wouldn't consider that a search engine bot, which to my knowledge is what robots.txt is trying to block.

But this may just be selective ignorance, because I might do it until the webmaster of that site contacted me and asked me to stop :)

Answer 3

"No" means "no".

Answer 4

An interesting IRL version of story involving The Harvard Coop: Coop Calls Cops On ISBN Copiers.