问题
Scrapy seems to complete without processing all the requests. I know this because i am logging before and after queueing the request and I can clearly see that.
I am logging in both parse and error callback methods and none of them got called for those missing requests.
How can I debug what happened to those requests?
回答1:
You need to add dont_filter=True when re-queueing the request. Though the request may not match other request but Scrapy remembers what requests it has already made and it will filter out if you re-queue it. It will assume it was by mistake.
来源：https://stackoverflow.com/questions/46466345/requests-disappear-after-queueing-in-scrapy