Read remote file using python subprocess and ssh?

Question

How can I read data from a big remote file using subprocess and ssh?

Answer 1

import subprocess
ssh = subprocess.Popen(['ssh', 'user@host', 'cat', 'path/to/file'],
                       stdout=subprocess.PIPE)
for line in ssh.stdout:
    line  # do stuff

Answer 2

For performance improvement, which is important when the file is big, there is rsync. For more information about the exact improvement see following post and the answer from Rafa: How does `scp` differ from `rsync`?

The algorithm would then be the following using rsync

import subprocess

subprocess.Popen(["rsync", host-ip+'/path/to/file'],stdout=subprocess.PIPE)
for line in ssh.stdout:
    line  # do stuff

Answer 3

The answer above will work, but you'll have to setup your ssh login to use no password between your boxes. There are other ways to transfer files between computers using Python. A simple way, without authentication is to setup an apache server and use an http request.

Answer 4

Use iter with readline to read each full line:

for i in iter(f.stdout.readline,"")