Getting progress message from a subprocess

Question

I want to start a program which needs several minutes to complete. During this time I want to read the progress message of the program (which are printed on the stdout). The pro

Answer 1

It's certainly possible: my package python-gnupg does exactly this, spawning gpg (Gnu Privacy Guard) under a subprocess. In the general case you need to specify subprocess.PIPE for the subprocess stdout and stderr; then create two separate threads which read the subprocess stdout and stderr to wherever you like.

In the case of python-gnupg, status messages from gpg are read and acted upon while the gpg process is running (not waiting until it's finished).

Basically, pseudocode is

process = subprocess.Popen(..., stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stderr = process.stderr
rr = threading.Thread(target=response_reader_func, args=(process.stderr,))
rr.setDaemon(True)
rr.start()

dr = threading.Thread(target=data_reader_func, args=(process.stdout,))
dr.setDaemon(True)
dr.start()

dr.join()
rr.join()
process.wait()

The reader functions are typically methods of an enclosing class which do the right thing based on what they're reading (in your case, updating progress info in some way).

Answer 2

You can do a poll on the status of your subprocess and keep outputting lines.

p = subprocess.Popen('ls;sleep 10', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

rc = p.poll()
while rc != 0:
    while True:
        line = p.stdout.readline()
        if not line:
            break
        print line
    rc = p.poll()

assert rc == 0

Answer 3

Simplest is to call Popen with the keyword argument stdout=subprocess.PIPE.

p = subprocess.Popen(["ls"], stdout=subprocess.PIPE)
while True:
    line = p.stdout.readline()
    if not line:
        break
    print line

---

To see this in action, here are two sample scripts. Make them both in the same directory and run python superprint.py

printandwait.py:

import time
import sys
print 10
sys.stdout.flush()
time.sleep(10)
print 20
sys.stdout.flush()

superprint.py:

import subprocess
import sys
p = subprocess.Popen(["python printandwait.py"], shell=True, stdout=subprocess.PIPE)
while True:
    print "Looping"
    line = p.stdout.readline()
    if not line:
        break
    print line.strip()
    sys.stdout.flush()