Add a delay after 500 requests scrapy

Question

I have a list of start 2000 urls and I\'m using:

DOWNLOAD_DELAY = 0.25 

For controlling the speed of the requests, But I also want to add a

Answer 1

You can look into using an AutoThrottle extension which does not give you a tight control of the delays but instead has its own algorithm of slowing down the spider adjusting it on the fly depending on the response time and number of concurrent requests.

If you need more control over the delays at certain stages of the scraping process, you might need a custom middleware or a custom extension (similar to AutoThrottle - source).

You can also change the .download_delay attribute of your spider on the fly. By the way, this is exactly what AutoThrottle extension does under-the-hood - it updates the .download_delay value on the fly.

Some related topics:

• Per request delay
• Request delay configurable for each Request