Python Popen - wait vs communicate vs CalledProcessError

Question

Continuing from my previous question I see that to get the error code of a process I spawned via Popen in python I have to call either wait() or communicate() (which can be

Answer 1

• about the deadlock: It is safe to use stdout=PIPE and wait() together iff you read from the pipe. .communicate() does the reading and calls wait() for you
• about the memory: if the output can be unlimited then you should not use .communicate() that accumulates all output in memory.

what is the proper thing to use here ?

To start subprocess, read its output line by line and to wait for it to exit:

#!/usr/bin/env python
from subprocess import Popen, PIPE

process = Popen(command, stdout=PIPE, bufsize=1)
with process.stdout:
    for line in iter(process.stdout.readline, b''): 
        handle(line)
returncode = process.wait() 

This code does not deadlock due to a finite OS pipe buffer. Also, the code supports commands with unlimited output (if an individual line fits in memory).

iter() is used to read a line as soon as the subprocess' stdout buffer is flushed, to workaround the read-ahead bug in Python 2. You could use a simple for line in process.stdout if you don't need to read lines as soon as they are written without waiting for the buffer to fill or the child process to end. See Python: read streaming input from subprocess.communicate().

If you know that the command output can fit in memory in all cases then you could get the output all at once:

#!/usr/bin/env python
from subprocess import check_output

all_output = check_output(command)

It raises CalledProcessError if the command returns with a non-zero exit status. Internally, check_output() uses Popen() and .communicate()

There should be one-- and preferably only one --obvious way to do it

subprocess.Popen() is the main API that works in many many cases. There are convenience functions/methods such as Popen.communicate(), check_output(), check_call() for common use-cases.

There are multiple methods, functions because there are multiple different use-cases.