Scrapy CrawlSpider retry scrape

Question

For a page that I\'m trying to scrape, I sometimes get a \"placeholder\" page back in my response that contains some javascript that autoreloads until it gets the real page.

Answer 1

I would think about having a custom Retry Middleware instead - similar to a built-in one.

Sample implementation (not tested):

import logging

logger = logging.getLogger(__name__)


class RetryMiddleware(object):
    def process_response(self, request, response, spider):
        if 'var PageIsLoaded = false;' in response.body:
            logger.warning('parse_page encountered an incomplete rendering of {}'.format(response.url))
            return self._retry(request) or response

        return response

    def _retry(self, request):
        logger.debug("Retrying %(request)s", {'request': request})

        retryreq = request.copy()
        retryreq.dont_filter = True
        return retryreq

And don't forget to activate it.