Python PIPE to popen stdin

Question

I am attempting something very similar to real time subprocess.Popen via stdout and PIPE

I, however, want to send input to the running process as well.

If I

Answer 1

First, you obviously need to add stdin=subprocess.PIPE to the Popen constructor, and then you can process.stdin.write just as you process.stdout.read.

But obviously, just as read can block if there's no data yet, write can block if the child isn't reading.

And even beyond the obvious, it's actually very hard to get the details right for using PIPEs in both directions with Popen to an interactive program without blocking anywhere. If you really want to do it, look at the source for communicate to see how it works. (There are known bugs before 3.2, so if you're on 2.x, you may have to do some backporting.) You will have to implement the code yourself, and if you want it to be cross-platform, you're going to have to do the whole mess that communicate does internally (spawning reader and writer threads for the pipes, etc.), and of course add another thread to not block the main thread on each attempt to communicate, and some kind of mechanism to message the main thread when the child is ready, and so on.

Alternatively, you can look at the various "async subprocess" projects on PyPI. The simplest one I know of today is async_subprocess, which basically just gives you a communicate that you can use without blocking.

Or, if you can use twisted (or possibly other event-based networking frameworks), there are wrappers around subprocess that plug into its event loop. (If you can wait for Python 3.4, or use the work-in-progress tulip on 3.3, someone's built something similar around tulip that may make it into 3.4.) And twisted even knows how to plug into Tkinter, so you don't have to manually handle two separate event loops and communicate between them.

If you only care about modern POSIX systems (not Windows), you can make it simpler by just putting the pipes in non-blocking mode and writing your code as if you were dealing with sockets.

But the easiest solution is probably to use something like pexpect instead of trying to script it manually. (As J.F. Sebastian points out, pexpect is Unix-only, but you can use a wrapper around pexpect for Unix and winpexpect for Windows.)