We are trying to crawl the web with our spider (respecting robots.txt etc) . The problem we are having, is that we keep getting hit with reports at abuseat.org due to hitting ma
