How to block a bot that is excessively visiting my site?

Question

This bot doesn\'t respect nofollow noindex in robots.txt.

I have this in robots.txt:

User-agent: Msnbot
Disallow: /

User-Agent: Msnbot/2.0b         


        

Answer 1

Here's what you need to do instead:

Code:

User-agent: *
Disallow:

User-agent: MSNbot
Disallow: /

The above code allows all robots except MSNbot.

You can read more about the robots exclusion protocol here.

for example, for bing.

User-agent: MSNBot
Disallow: /

for google

User-agent: googlebot

Disallow: /

if you want block all bots. use this.

User-agent: *

Disallow: /

Answer 2

Based on Block by useragent or empty referer you could something like this in your .htaccess

Options +FollowSymlinks  
RewriteEngine On  
RewriteBase /  
SetEnvIfNoCase User-Agent "^Msnbot" ban_agent
Deny from env=ban_agent