Scrapy CLOSESPIDER_PAGECOUNT setting don't work as should

Question

I use scrapy 1.0.3 and can\'t discover how works CLOSESPIDER extesnion. For command: scrapy crawl domain_links --set=CLOSESPIDER_PAGECOUNT=1 is correctly one requst, but for

Answer 1

CLOSESPIDER_PAGECOUNT is controlled by the CloseSpider extension, which counts every response until reaching its limit that's when it tells the crawler process to start ending (finishing requests and closing available slots).

Now the reason why your spider ends when you specify CLOSESPIDER_PAGECOUNT=1 is because at that moment (when it gets its first response) there are no pending requests, they are being created after your first one, so the crawler process is ready to end, not taking into account the following ones (because they will be born after the first).

When you specify CLOSESPIDER_PAGECOUNT>1, your spider is caught creating requests and filling the requests queue. When the spider knows when to finish there are still pending requests to process, which are executed as part of closing the spider.