Copying file from HDFS to Local Machine

Question

I\'m having a problem with trying to \"download\" file from HDFS file system to my local system. (even though opposite operation works without a problem). *Note: File exists on

Answer 1

Try using this method from the API :

//where delSrc is do you want to delete the source, src and dst you already have and useRawLocalFileSystem should be set to true in your case
hdfsFileSystem.copyToLocalFile(delSrc, src, dst, useRawLocalFileSystem);

in your case replace the :

hdfsFileSystem.copyToLocalFile(hdfs, local);

with:

hdfsFileSystem.copyToLocalFile(false, hdfs, local, true);

Answer 2

You can follow code shown below:

public static void main(String args[]){
    try {
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://localhost:54310/user/hadoop/");
        FileSystem fs = FileSystem.get(conf);
        FileStatus[] status = fs.listStatus(new Path("hdfsdirectory"));
        for(int i=0;i<status.length;i++){
            System.out.println(status[i].getPath());
            fs.copyToLocalFile(false, status[i].getPath(), new Path("localdir"));
        }
    } catch (IOException e) {
        e.printStackTrace();
    }

}