Is it possible to save files in Hadoop without saving them in local file system?

Question

Is it possible to save files in Hadoop without saving them in local file system? I would like to do something like shown below however I would like to save file directly in

Answer 1

Hadoop has REST APIs that allow you to create files via WebHDFS.

So you could write your own create based on the REST APIs using a python library like requests for doing the HTTP. However, there are also several python libraries that support Hadoop/HDFS and already use the REST APIs or that use the RPC mechanism via libhdfs.

• pydoop
• hadoopy
• snakebite
• pywebhdfs
• hdfscli
• pyarrow

Just make sure you look for how to create a file rather than having the python library call hdfs dfs -put or hadoop fs -put.

See the following for more information:

• pydoop vs hadoopy - hadoop python client
• List all files in HDFS Python without pydoop
• A Guide to Python Frameworks for Hadoop
• Native Hadoop file system (HDFS) connectivity in Python
• PyArrow
• https://github.com/pywebhdfs/pywebhdfs
• https://github.com/spotify/snakebite
• https://crs4.github.io/pydoop/api_docs/hdfs_api.html
• https://hdfscli.readthedocs.io/en/latest/
• WebHDFS REST API:Create and Write to a File