save url as a file name in python

Question

Firstly, I\'m pretty new in python, please leave a comment as well if you consider to down vote

I have a url such as

http://example.com/here/there/         


        

Answer 1

You have several problems. One of them is that Unix shell abbreviations (~) are not going to be auto-interpreted by Python as they are in Unix shells.

The second is that you're not going to have good luck writing a file path in Unix that has embedded slashes. You will need to convert them to something else if you're going to have any luck of retrieving them later. You could do that with something as simple as response.url.replace('/','_'), but that will leave you with many other characters that are also potentially problematic. You may wish to "sanitize" all of them on one shot. For example:

import os
import urllib

def write_response(response, filedir='~'):
    filedir = os.path.expanduser(dir)
    filename = urllib.quote(response.url, '')
    filepath = os.path.join(filedir, filename)
    with open(filepath, "w") as f:
        f.write(response.body)

This uses os.path functions to clean up the file paths, and urllib.quote to sanitize the URL into something that could work for a file name. There is a corresponding unquote to reverse that process.

Finally, when you write to a file, you may need to tweak that a bit depending on what the responses are, and how you want them written. If you want them written in binary, you'll need "wb" not just "w" as the file mode. Or if it's text, it might need some sort of encoding first (e.g., to utf-8). It depends on what your responses are, and how they are encoded.