Proper way of re-using and closing a subprocess object

Question

I have the following code in a loop:

while true:
    # Define shell_command
    p1 = Popen(shell_command, shell=shell_type, stdout=PIPE, stderr=PIPE, preexec         


        

Answer 1

Considering the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands?

Yes.

#!/usr/bin/env python
from __future__ import print_function
import uuid
import random
from subprocess import Popen, PIPE, STDOUT

MARKER = str(uuid.uuid4())

shell_command = 'echo a'
p = Popen('sh', stdin=PIPE, stdout=PIPE, stderr=STDOUT,
          universal_newlines=True) # decode output as utf-8, newline is '\n'
while True:
    # write next command
    print(shell_command, file=p.stdin)
    # insert MARKER into stdout to separate output from different shell_command
    print("echo '%s'" % MARKER, file=p.stdin)
    # read command output
    for line in iter(p.stdout.readline, MARKER+'\n'):
        if line.endswith(MARKER+'\n'):
            print(line[:-len(MARKER)-1])
            break # command output ended without a newline
        print(line, end='')
    # exit on condition
    if random.random() < 0.1:
        break
# cleanup
p.stdout.close()
if p.stderr:
   p.stderr.close()
p.stdin.close()
p.wait()

Put while True inside try: ... finally: to perform the cleanup in case of exceptions. On Python 3.2+ you could use with Popen(...): instead.

Would that be more efficient in any way than opening new subprocess objects each time?

Does it matter in your case? Don't guess. Measure it.

Answer 2

The "correct" order is:

1. Create a thread to read stdout (and a second one to read stderr, unless you merged them into one).

2. Write commands to be executed by the child to stdin. If you're not reading stdout at the same time, writing to stdin can block.

3. Close stdin (this is the signal for the child that it can now terminate by itself whenever it is done)

4. When stdout returns EOF, the child has terminated. Note that you need to synchronize the stdout reader thread and your main thread.

5. call wait() to see if there was a problem and to clean up the child process

If you need to stop the child process for any reason (maybe the user wants to quit), then you can:

1. Close stdin if the child terminates when it reads EOF.

2. Kill the with terminate(). This is the correct solution for child processes which ignore stdin.

3. If the child doesn't respond, try kill()

In all three cases, you must call wait() to clean up the dead child process.

Answer 3

What is the proper way of closing a subprocess object once we are done using it?

stdout.close() and stdin.close() will not terminate a process unless it exits itself on end of input or on write errors.

.terminate() and .kill() both do the job, with kill being a bit more "drastic" on POSIX systems, as SIGKILL is sent, which cannot be ignored by the application. Specific differences are explained in this blog post, for example. On Windows, there's no difference.

Also, remember to .wait() and to close the pipes after killing a process to avoid zombies and force the freeing of resources.

A special case that is often encountered are processes which read from STDIN and write their result to STDOUT, closing themselves when EOF is encountered. With these kinds of programs, it's often sensible to use subprocess.communicate:

>>> p = Popen(["sort"], stdin=PIPE, stdout=PIPE)
>>> p.communicate("4\n3\n1")
('1\n3\n4\n', None)
>>> p.returncode
0

This can also be used for programs which print something and exit right after:

>>> p = Popen(["ls", "/home/niklas/test"], stdin=PIPE, stdout=PIPE)
>>> p.communicate()
('file1\nfile2\n', None)
>>> p.returncode
0

Considering the nature of my script, is there a way to open a subprocess object only once and reuse it with different shell commands? Would that be more efficient in any way than opening new subprocess objects each time?

I don't think the subprocess module supports this and I don't see what resources could be shared here, so I don't think it would give you a significant advantage.

Answer 4

1. Depends on what you expect the process to do; you should always call p1.wait() in order to avoid zombies. Other steps depend on the behaviour of the subprocess; if it produces any output, you should consume the output (e.g. p1.read() ...but this would eat lots of memory) and only then call the p1.wait(); or you may wait for some timeout and call p1.terminate() to kill the process if you think it doesn't work as expected, and possible call p1.wait() to clean the zombie.

Alternatively, p1.communicate(...) would do the handling if io and waiting for you (not the killing).

1. Subprocess objects aren't supposed to be reused.