Scrapy Crawling Speed is Slow (60 pages / min)

Question

I am experiencing slow crawl speeds with scrapy (around 1 page / sec). I\'m crawling a major website from aws servers so I don\'t think its a network issue. Cpu utilization is n

Answer 1

Are you sure you are allowed to crawl the destination site at high speed? Many sites implement download threshold and "after a while" start responding slowly.