How to use python requests with scrapy?

為{幸葍}努か 提交于 2021-01-21 11:55:39

问题


I am trying to use requests to fetch a page then pass the response object to a parser, but I ran into a problem:

def start_requests(self):
    yield self.parse(requests.get(url))
def parse(self, response):
  #pass

builtins.AttributeError: 'generator' object has no attribute 'dont_filter'


回答1:


You first need to download the page's resopnse and then convert that string to HtmlResponse object

from scrapy.http import HtmlResponse
resp = requests.get(url)

response = HtmlResponse(url="", body=resp.text, encoding='utf-8')


来源:https://stackoverflow.com/questions/56230826/how-to-use-python-requests-with-scrapy

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!