How can I print and display subprocess stdout and stderr output without distortion?

Question

Maybe there\'s someone out in the ether that can help me with this one. (I have seen a number of similar questions to this on SO, but none deal with both standard out and st

Answer 1

When I tested it, it seemed readline() is blocking. However I was able to access stdout and stderr separately using threads. Code sample as follows:

import os
import sys
import subprocess
import threading

class printstd(threading.Thread):
    def __init__(self, std, printstring):
        threading.Thread.__init__(self)
        self.std = std
        self.printstring = printstring
    def run(self):
        while True:
          line = self.std.readline()
          if line != '':
            print self.printstring, line.rstrip()
          else:
            break

pythonfile = os.path.join(os.getcwd(), 'mypythonfile.py')

process = subprocess.Popen([sys.executable,'-u',pythonfile], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

print 'Process ID:', process.pid

thread1 = printstd(process.stdout, 'stdout:')
thread2 = printstd(process.stderr, 'stderr:')

thread1.start()
thread2.start()

threads = []

threads.append(thread1)
threads.append(thread2)

for t in threads:
    t.join()

However, I am not certain that this is thread-safe.

Answer 2

Combining this answer with this, the following code works for me:

import subprocess, sys
p = subprocess.Popen(args, stderr=sys.stdout.fileno(), stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ""):
 print line,

Answer 3

Make the pipes non-blocking by using fcntl.fcntl, and use select.select to wait for data to become available in either pipe. For example:

# Helper function to add the O_NONBLOCK flag to a file descriptor
def make_async(fd):
    fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)

# Helper function to read some data from a file descriptor, ignoring EAGAIN errors
def read_async(fd):
    try:
        return fd.read()
    except IOError, e:
        if e.errno != errno.EAGAIN:
            raise e
        else:
            return ''

process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
make_async(process.stdout)
make_async(process.stderr)

stdout = str()
stderr = str()
returnCode = None

while True:
    # Wait for data to become available 
    select.select([process.stdout, process.stderr], [], [])

    # Try reading some data from each
    stdoutPiece = read_async(process.stdout)
    stderrPiece = read_async(process.stderr)

    if stdoutPiece:
        print stdoutPiece,
    if stderrPiece:
        print stderrPiece,

    stdout += stdoutPiece
    stderr += stderrPiece
    returnCode = process.poll()

    if returnCode != None:
        return (returnCode, stdout, stderr)

Note that fcntl is only available on Unix-like platforms, including Cygwin.

If you need it to work on Windows without Cygwin, it's doable, but it's much, much tougher. You'll have to:

• Use the pywin32 library to call through to the native Win32 API
• Use SetNamedPipeHandleState with PIPE_NOWAIT to make the stdout and stderr pipes non-blocking
• Use WaitForMultipleObjects instead of select to wait for data to become available
• Use ReadFile to read the data