Scrapy: non-blocking pause

Question

I have a problem. I need to stop the execution of a function for a while, but not stop the implementation of parsing as a whole. That is, I need a non-blocking pause.

It

Answer 1

If you're attempting to use this for rate limiting, you probably just want to use DOWNLOAD_DELAY instead.

Scrapy is just a framework on top of Twisted. For the most part, you can treat it the same as any other twisted app. Instead of calling sleep, just return the next request to make and tell twisted to wait a bit. Ex:

from twisted.internet import reactor, defer

def non_stop_function(self, response)
    d = defer.Deferred()
    reactor.callLater(10.0, d.callback, Request(
        'some url',
        callback=self.non_stop_function
    ))
    return d