Is it possible to save files in Hadoop without saving them in local file system?

Question

Is it possible to save files in Hadoop without saving them in local file system? I would like to do something like shown below however I would like to save file directly in

Answer 1

Here's how to download a file directly to HDFS with Pydoop:

import os
import requests
import pydoop.hdfs as hdfs


def dl_to_hdfs(url, hdfs_path):
    r = requests.get(url, stream=True)
    with hdfs.open(hdfs_path, 'w') as f:
        for chunk in r.iter_content(chunk_size=1024):
            f.write(chunk)


URL = "https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz"
dl_to_hdfs(URL, os.path.basename(URL))

The above snippet works for a generic URL. If you already have the file as a Django UploadedFile, you can probably use its .chunks method to iterate through the data.