I have just stumbled upon Bad Behavior - a plugin for PHP that promises to detect spam and malicious crawlers by preventing them from accessing the site at all.
Does something similar exist for ASP.NET and ASP.NET MVC?
I am interested in blocking access to the site altogether, not in detecting spam after it was posted.
EDIT: I am interested specifically in solutions that will detect access patterns to the site - these would prevent screen scraping the site as a whole, or at least make it a very slow process for the offender because the bot would have to act as a regular user in frequency of his clicks and in patterns found in the clicks (100 links visited in depth-first succession are a clear indication that someone is crawling the site).
By malicious crawler I mean primarily bots that are building duplicate copies of other sites but also spambots that are trying to locate some submittable forms on the site and submit their crap.
The NoBot control in the ASP.NET AJAX Toolkit might help, if I understand you correctly. From the description page:
NoBot is a control that attempts to provide CAPTCHA-like bot/spam prevention without requiring any user interaction. This approach is easier to bypass than an implementation that requires actual human intervention, but NoBot has the benefit of being completely invisible.
I haven't used this one myself, so I can't speak to its effectiveness against sophisticated bots, but it sounds like it's meant to be exactly what you're talking about. It has the benefit of being very simple; essentially a drop-in you can add to a master page.
来源：https://stackoverflow.com/questions/2667734/malicious-crawler-blocker-for-asp-net