how to kill (or avoid) zombie processes with subprocess module

Question

When I kick off a python script from within another python script using the subprocess module, a zombie process is created when the subprocess \"completes\". I am unable to

Answer 1

I'm not entirely sure what you mean by no_wait(). Do you mean you can't block waiting for child processes to finish? Assuming so, I think this will do what you want:

os.wait3(os.WNOHANG)

Answer 2

If you simply use subprocess.Popen, you'll be fine - here's how:

import subprocess

def spawn_some_children():
    subprocess.Popen(["sleep", "3"])
    subprocess.Popen(["sleep", "3"])
    subprocess.Popen(["sleep", "3"])

def do_some_stuff():
    spawn_some_children()
    # do some stuff
    print "children went out to play, now I can do my job..."
    # do more stuff

if __name__ == '__main__':
    do_some_stuff()

You can use .poll() on the object returned by Popen to check whether it finished (without waiting). If it returns None, the child is still running.

Make sure you don't keep references to the Popen objects - if you do, they will not be garbage collected, so you end up with zombies. Here's an example:

import subprocess

def spawn_some_children():
    children = []
    children.append(subprocess.Popen(["sleep", "3"]))
    children.append(subprocess.Popen(["sleep", "3"]))
    children.append(subprocess.Popen(["sleep", "3"]))
    return children

def do_some_stuff():
    children = spawn_some_children()
    # do some stuff
    print "children went out to play, now I can do my job..."
    # do more stuff

    # if children finish while we are in this function,
    # they will become zombies - because we keep a reference to them

In the above example, if you want to get rid of the zombies, you can either .wait() on each of the children or .poll() until the result is not None.

Either way is fine - either not keeping references, or using .wait() or .poll().

Answer 3

Recently, I came across this zombie problem due to my python script. The actual problem was mainly due to killing of the subprocess and the parent process doesn't know that the child is dead. So what I did was, just adding popen.communicate() after the kill signal of child process so that the parent process comes to know that the child is dead, then the kernel updates the pid of the childprocess since the child is no more and so there is no zombies formed now.

PS:poll is also an option here since it checks and conveys about the child status to the parent. Often in subprocess it is better if u use check_output or call if u need not communicate with the stdout and stdin.

Answer 4

Like this:
s = Popen(args)
s.terminate()
time.sleep(0.5)
s.poll()

it works
zombie processes will disappear

Answer 5

If you delete the subprocess object, using del to force garbage collection, that will cause the subprocess object to be deleted and then the defunct processes will go away without terminating your interpreter. You can try this out in the python command line interface first.

Answer 6

Not using Popen.communicate() or call() will result in a zombie process.

If you don't need the output of the command, you can use subprocess.call():

>>> import subprocess
>>> subprocess.call(['grep', 'jdoe', '/etc/passwd'])
0

If the output is important, you should use Popen() and communicate() to get the stdout and stderr.

>>> from subprocess import Popen, PIPE
>>> process = Popen(['ls', '-l', '/tmp'], stdout=PIPE, stderr=PIPE)
>>> stdout, stderr = process.communicate()
>>> stderr
''
>>> print stdout
total 0
-rw-r--r-- 1 jdoe jdoe 0 2010-05-03 17:05 bar
-rw-r--r-- 1 jdoe jdoe 0 2010-05-03 17:05 baz
-rw-r--r-- 1 jdoe jdoe 0 2010-05-03 17:05 foo