Using subprocess with select and pty hangs when capturing output

Question

I\'m trying to write a python program that is able to interact with other programs. That means sending stdin and receiving stdout data. I cannot use pexpect (although it definit

Answer 1

From what I understand, you do not need to use pty. runner.py can be modified as

import subprocess
import sys

def main():
        process = subprocess.Popen(['python', 'outputter.py'],
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE)

        while process.poll() is None:
                output = process.stdout.readline()
                sys.stdout.write(output)
                sys.stdout.flush()
        print "**ALL COMPLETED**"

if __name__ == '__main__':
        main()

process.stdout.read(1) can be used instead of process.stdout.readline() for real-time output per character from the subprocess.

Note: If you do not require real-time output from the subprocess, use Popen.communicate to avoid the polling loop.

Answer 2

When your child process exits - your parent process gets SIGCHLD signal. By default this signal is ignored but you can intercept it:

import sys
import signal

def handler(signum, frame):
    print 'Child has exited!'
    sys.exit(0)

signal.signal(signal.SIGCHLD, handler)

The signal should also break the blocking syscall to 'select' or 'read' (or whatever you are in) and let you do whatever you have to (cleanup, exit, etc.) in handler function.

Answer 3

First of all, os.read does block, contrary to what you state. However, it does not block after select. Also os.read on a closed file descriptor always returns an empty string, that you might want to check for.

The real problem however is that the master device descriptor is never closed, thus the final select is the one that will block. In a rare race condition, the child process has exited between select and process.poll() and your program exits nicely. Most of the time however the select blocks forever.

If you install the signal handler as proposed by izhak all hell breaks loose; whenever a child process is terminated, the signal handler is run. After the signal handler is run, the original system call in that thread cannot be continued, so that syscall invocation returns nonzero errno, which often results in some random exception being thrown in python. Now, if elsewhere in your program you use some library with any blocking system calls that do not know how to handle such exceptions, you are in a big trouble (any os.read for example anywhere can now throw an exception, even after a successful select).

Weighing having random exceptions thrown anywhere against polling a bit, I don't think the timeout on select does not sound that bad idea. Your process would still hardly be the only (slow) polling process on the system anyway.

Answer 4

There are a number of things you can change to make your code correct. The simplest thing I can think of is just to close your parent process's copy of the slave fd after forking, so that when the child exits and closes its own slave fd, the parent's select.select() will mark the master as available for read, and the subsequent os.read() will give an empty result and your program will complete. (The pty master won't see the slave end as being closed until both copies of the slave fd are closed.)

So, just one line:

os.close(slave)

..placed immediately after the subprocess.Popen call, ought to fix your problem.

However, there are possibly better answers, depending on exactly what your requirements are. As someone else noted, you don't need a pty just to avoid buffering. You could use a bare os.pipe() in place of pty.openpty() (and treat the return value exactly the same). A bare OS pipe will never buffer; if the child process isn't buffering its output, then your select() and os.read() calls won't see buffering either. You would still need the os.close(slave) line, though.

But it's possible that you do need a pty for different reasons. If some of your child programs expect to be run interactively much of the time, then they might be checking to see if their stdin is a pty and behaving differently depending on the answer (lots of common utilities do this). If you really do want the child to think it has a terminal allocated for it, then the pty module is the way to go. Depending on how you'll run runner.py, you may need to switch from using subprocess to pty.fork(), so that the child has its session ID set and the pty pre-opened (or see the source for pty.py to see what it does and duplicate the appropriate parts in your subprocess object's preexec_fn callable).