Python subprocess call hangs

Question

Python version: 2.6.7 I have the following subprocess.call within a for loop which is exectuted 18 times, however, the process constantly hangs on the 19th

Answer 1

When using subprocess, I tend to do something like this:

is_running = lambda: my_process.poll() is None

my_process = subprocess.Popen(' '.join(exclude), 
                              stdout=subprocess.PIPE, 
                              stderr=subprocess.PIPE,
                              shell=True)

# Grab all the output from stdout and stderr and log it
while is_running():
    rlist, wlist, xlist = select.select([my_process.stdout, my_process.stderr], [], [], 1)

# Log stdout, but don't spam the log
if my_process.stdout in rlist and verbose:
    # Adjust the number of bytes read however you like, 1024 seems to work 
    # pretty well for me. 
    Tracer.log.debug(my_process.stdout.read(1024))

# Log stderr, always
if my_process.stderr in rlist:
    # Same as with stdout, adjust the bytes read as needed.
    Tracer.log.error(my_process.stderr.read(1024))

I've seen the stdout stuff just dump a bunch of empty lines in my logs in the past, which is why I log that at the debug level. That prints to my logs during the development, but never gets written in production, so I can safely leave it in the code for debugging without putting garbage in their logs.

Hopefully, this can help expose just where your program is hanging and what's causing it.