How to use Java JSch library to read remote file line by line?

Question

I am trying to use Java to read a file line by line, which is very simple (there are multiple solutions for this on stackoverflow.com), but the caveat here is that the file is located on a remote server, and it is not possible to get a local copy (it is a massive collection of millions of Amazon reviews in a single .txt file).

JSch comes with two example classes that copy files to and from remote hosts, namely ScpTo and ScpFrom. I am interested in reading the file from the remote host line by line; ScpFrom would try to copy the whole thing into a local file, which would take ages.

Here is a link to ScpFrom: http://www.jcraft.com/jsch/examples/ScpFrom.java.html

I would try to cargo cult the code there and then modify it to read a remote file line by line rather than write to a local file, but most of the code is Greek to me once the author declares a byte array and starts reading bytes from the remote file. I'll admit this is something I have almost no understanding of; BufferedReader provides a much higher level interface. Essentially I want to do this: How to read a large text file line by line using Java?

except using a BufferReader that can also read remote files line by line, if provided the host name and user credentials (password, etc.), i.e. RemoteBufferReader?

This is the test code I've written; how do I read in the remote file line by line using JSCh?

public class test2
 {
    static String user = "myusername";
    static String host = "user@remotehost";
    static String password = "mypasswd";
    static String rfile = "/path/to/remote/file/on/remote/host";
    public static void main(String[] args) throws FileNotFoundException, IOException, JSchException
    {
        JSch jsch=new JSch();
        Session session=jsch.getSession(user, host, 22);
        session.setPassword(password);
        session.connect();
        // exec 'scp -f rfile' remotely
        String command="scp -f "+rfile;
        Channel channel=session.openChannel("exec");
        ((ChannelExec)channel).setCommand(command);

        // get I/O streams for remote scp
        OutputStream out=channel.getOutputStream();
        channel.connect()
        //no idea what to do next

    }
 }

Answer 1

To manipulate files through ssh, you're better off using sftp than scp or pure ssh. Jsch has built-in support for sftp. Once you've opened a session, do this to open an sftp channel:

ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");

Once you've opened an sftp channel, there are methods to read a remote file which let you access the file's content as an InputStream. You can convert that to a Reader if you need to read line-by-line:

InputStream stream = sftp.get("/some/file");
try {
    BufferedReader br = new BufferedReader(new InputStreamReader(stream));
    // read from br
} finally {
    stream.close();
}

Using try with resources syntax, your code might look more like this:

try (InputStream is = sftp.get("/some/file");
     InputStreamReader isr = new InputStreamReader(stream);
     BufferedReader br = new BufferedReader(isr)) {
    // read from br
}

Answer 2

JSch library is the powerful library that can be used to read file from SFTP server. Below is the tested code to read file from SFTP location line by line

        JSch jsch = new JSch();
        Session session = null;
        try {
            session = jsch.getSession("user", "127.0.0.1", 22);
            session.setConfig("StrictHostKeyChecking", "no");
            session.setPassword("password");
            session.connect();

            Channel channel = session.openChannel("sftp");
            channel.connect();
            ChannelSftp sftpChannel = (ChannelSftp) channel;

            InputStream stream = sftpChannel.get("/usr/home/testfile.txt");
            try {
                BufferedReader br = new BufferedReader(new InputStreamReader(stream));
                String line;
                while ((line = br.readLine()) != null) {
                    System.out.println(line);
                }

            } catch (IOException io) {
                System.out.println("Exception occurred during reading file from SFTP server due to " + io.getMessage());
                io.getMessage();

            } catch (Exception e) {
                System.out.println("Exception occurred during reading file from SFTP server due to " + e.getMessage());
                e.getMessage();

            }

            sftpChannel.exit();
            session.disconnect();
        } catch (JSchException e) {
            e.printStackTrace();
        } catch (SftpException e) {
            e.printStackTrace();
        }

Please refer the blog for whole program.