I\'ve been undertaking some fairly large scraping projects (>10000 pages), and some of the websites I am getting information from ask for a crawl delay of 10 seconds. However