How do you open a file stream for reading using Scrapy?

问题

Using Scrapy, I want to use my extracted url to read a binary file into memory and extract the contents.

Currently, I can find the URL on the page using a selector e.g.

myFile = response.xpath('//a[contains(@href,".interestingfileextension")]/@href').extract()

How do I then read that file into memory so that I can look for content in that file?

Many thanks

回答1:

Make a request and explore the content in the callback:

def parse(self, response):
    url = response.xpath('//a[contains(@href,".interestingfileextension")]/@href').extract_first()
    return scrapy.Request(url, callback=self.parse_file)

def parse_file(self, response):
    # response here is the contents of the file
    print(response.body)

来源：https://stackoverflow.com/questions/36226727/how-do-you-open-a-file-stream-for-reading-using-scrapy