How to use python requests with scrapy?

问题

I am trying to use requests to fetch a page then pass the response object to a parser, but I ran into a problem:

def start_requests(self):
    yield self.parse(requests.get(url))
def parse(self, response):
  #pass

builtins.AttributeError: 'generator' object has no attribute 'dont_filter'

回答1:

You first need to download the page's resopnse and then convert that string to HtmlResponse object

from scrapy.http import HtmlResponse
resp = requests.get(url)

response = HtmlResponse(url="", body=resp.text, encoding='utf-8')

来源：https://stackoverflow.com/questions/56230826/how-to-use-python-requests-with-scrapy