Scrapy - set delay to retry middleware

Question

I\'m using Scrapy-splash and I have a problem with memory. I can clearly see that memory used by docker python3 is gradually increasin

Answer 1

1. A more elaborate solution could be to set up a Kubernetes cluster in which you have multiple replicas running. This way you avoid having a failure of just 1 container impacting your scraping job.

2. I don't think it's easy to configure a waiting time only for retries. You could play with DOWNLOAD_DELAY (but this will impact delay between all requests), or set the RETRY_TIMES to a higher value than the default of 2.