Scrapy CrawlSpider + Splash: how to follow links through linkextractor?

Question

I have the following code that is partially working,

class ThreadSpider(CrawlSpider):
    name = \'thread\'
    allowed_domains = [\'bbs.example.com\']
    star         


        

Answer 1

Seems to be related to https://github.com/scrapy-plugins/scrapy-splash/issues/92

Personnaly I use dont_process_response=True so response is HtmlResponse (which is required by the code in _request_to_follows).

And I also redefine the _build_request method in my spyder, like so:

def _build_request(self, rule, link):
    r = SplashRequest(url=link.url, callback=self._response_downloaded, args={'wait': 0.5}, dont_process_response=True)
    r.meta.update(rule=rule, link_text=link.text)
    return r 

In the github issues, some users just redefine the _request_to_follow method in their class.