WEBHDFS REST API to copy/move files from windows server/local folder/desktop to HDFS

问题

Using WEBHDFS REST API calls can i transfer or copy the files from Windows machine(i.e. windows server or windows local folder or desktop) to Hadoop-HDFS file system?

If yes any sample command info?

I have tried and i was able to do using
Windows->(using ftp)-> Linux directory -> (using webhdfs) -> HDFS and this is two step process and i am looking for one step process directly from Windows -> (webhdfs) -> HDFS.

I referred in https://hadoop.apache.org/docs/r1.0.4/webhdfs.html for helpful info also.

Example : if my file is in E:\user\accounts.txt and i want to move this file to HDFS /user/kumar/ folder using webhdfs.

Currently what i am doing is Step-1) ftp accounts file from Windows to linux directory.
Step-2) running curl commands to move the file from linux machine to HDFS folders.

Any suggestion to do it in one step process? Step-1) Windows -> HDFS using webhdfs in one step.

回答1:

We can copy files from windows file system to HDFS by using scp command.

scp source_file_name user@/path/file_name

and also we can achieve this by using winscp tool. you can install it and establish a connect to hdfs server then files can be transfer.

回答2:

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

import jcifs.smb.NtlmPasswordAuthentication;
import jcifs.smb.SmbFile;
import jcifs.smb.SmbFileInputStream;

public class FileWriteToHDFS 
{
    public static void main(String[] args) throws Exception 
    {
        String src = args[0];
        String dest = args[1];

        Console console = System.console();
        String username = console.readLine("Username: ");
        String password = new String(console.readPassword("Password: "));
        String domain = console.readLine("Authentication Domain: ");

        NtlmPasswordAuthentication auth = new NtlmPasswordAuthentication(domain, username, password);
        SmbFile srcSMB = new SmbFile (src, auth);

        InputStream in = new SmbFileInputStream(srcSMB);

        Configuration myConf = new Configuration();

        FileSystem fs = FileSystem.get(URI.create(dest), myConf);
        OutputStream out = fs.create(new Path(dest));
        try
        {
            IOUtils.copyBytes(in, out, 4096, false);
        }
        catch(IOException e)
        {
            e.printStackTrace();
        }
        finally
        {
            IOUtils.closeStream(in);
        }
    }
}

This code uses JCIFS to copy through smb protocol to HDFS

来源：https://stackoverflow.com/questions/38921725/webhdfs-rest-api-to-copy-move-files-from-windows-server-local-folder-desktop-to

标签

curl

HDFS

hadoop2

jira-rest-api

webhdfs