how to allow known web crawlers and block spammers and harmful robots from scanning asp.net website

Question

How can I configure my site to allow crawling from well known robots like google, bing, yahoo, alexa etc. and stop other harmful spammers, robots

should i block particul

Answer 1

I like to use the .htaccess file, once you have a list of known bots add these lines to the bottom of your file.

RewriteCond %{HTTP_REFERER} ^http(s)?://([^.]+.)suspectIP.$ [NC,OR]

RewriteCond %{HTTP_REFERER} ^http(s)?://([^.]+.)suspectURL.com.$ [NC]

RewriteRule (.*) - [F]