Use python subprocess module like a command line simulator

Question

I am writing a test framework in Python for a command line application. The application will create directories, call other shell scripts in the current directory and will outpu

Answer 1

Instead of using plain subprocess you might find Python sh library very useful:

http://amoffat.github.com/sh/

Here is an example how to build in an asynchronous interaction loop with sh:

http://amoffat.github.com/sh/tutorials/2-interacting_with_processes.html

Another (old) library for solving this problem is pexpect:

http://www.noah.org/wiki/pexpect

Answer 2

subprocess.Popen allows you to continue execution after starting a process. The Popen objects expose wait(), poll() and many other methods to communicate with a child process when it is running. Isn't it what you need?

See Popen constructor and Popen objects description for details.

Here is a small example that runs Bash on Unix systems and executes a command:

from subprocess import Popen, PIPE
p = Popen (['/bin/sh'], stdout=PIPE, stderr=PIPE, stdin=PIPE)
sout, serr = p.communicate('ls\n')
print 'OUT:'
print sout
print 'ERR:'
print serr

UPD: communicate() waits for process termination. If you do not need that, you may use the appropriate pipes directly, though that usually gives you rather ugly code.

UPD2: You updated the question. Yes, you cannot call communicate twice for a single process. You may either give all commands you need to execute in a single call to communicate and check the whole output, or work with pipes (Popen.stdin, Popen.stdout, Popen.stderr). If possible, I strongly recommend the first solution (using communicate).

Otherwise you will have to put a command to input and wait for some time for desired output. What you need is non-blocking read to avoid hanging when there is nothing to read. Here is a recipe how to emulate a non-blocking mode on pipes using threads. The code is ugly and strangely complicated for such a trivial purpose, but that's how it's done.

Another option could be using p.stdout.fileno() for select.select() call, but that won't work on Windows (on Windows select operates only on objects originating from WinSock). You may consider it if you are not on Windows.