Saving stdout from subprocess.Popen to file line by line

Question

My python script uses subprocess to call an another script, which produces output very slow(line-by-line basis). I would like to write the output line by line to file not when t

Answer 1

Thought I'd share a solution that doesn't use .poll(), .wait() or .communicate(). A couple of points:

• I use import codecs because my output includes East Asian UTF-8 text
• I trap each line with try: to filter out corrupted/invalid UTF-8 text
• I use '\x0a' to force Linux newline regardless of the platform.
• Use for line in iter(subproc.stderr.readline, ''): if you need to capture stderr
• This approach generates output only when child program creates output
• Using the kw dictionary is overkill for this example, but shows how to use **kwargs with subprocess

Code:

import subprocess
import codecs
import os

kw = {
    'bufsize': 0,
    'executable': None,
    'stdin': subprocess.PIPE,
    'stdout': subprocess.PIPE,
    'stderr': subprocess.PIPE,
    'preexec_fn': None,
    'close_fds': False,
    'shell': False,
    'cwd': None,
    'env': None,
    'universal_newlines': False,
    'startupinfo': None,
    'creationflags': 0,
    }

args = ['ls', '-lart']
kw['cwd'] = os.path.expanduser('~')
logfile = os.path.expanduser('~/stdout.txt')
stdlog = []

try:
    subproc = subprocess.Popen(args,**kw)
except:
    print 'Error loading subprocess. Check arguments and kwargs'
    exit()

log = codecs.open(logfile,'w','utf-8')
log.write(': Starting log for: \"%s\"\x0a'%(' '.join(args)))
for line in iter(subproc.stdout.readline, ''):
    try:
        stdlog.append(line.rstrip().decode('utf-8'))
        log.write(stdout[-1]+'\x0a')
        print stdout[-1]
    except:
        pass

log.flush()
log.close()