Access standard output of a sub process in python

Question

How can I access the stdout of child processes prior to sending them to main process? I am using multiprocessing.Pool module to generate child process pools.

Answer 1

The main process and the children all share the same standard input and standard output file descriptors. They have no control over what the other writes to them. The only thing you can do is replace stdin and stdout in the children with something else that the main process can control. As an example you could subclass a dummy file object like StringIO and redirect the data that the children write to this object to the parent via a Queue:

import sys
from multiprocessing import Queue, Pool, current_process
from StringIO import StringIO

class MyStringIO(StringIO):
    def __init__(self, queue, *args, **kwargs):
        StringIO.__init__(self, *args, **kwargs)
        self.queue = queue
    def flush(self):
        self.queue.put((current_process().name, self.getvalue()))
        self.truncate(0)

def initializer(queue):
     sys.stderr = sys.stdout = MyStringIO(queue)

def task(num):
     print num
     sys.stdout.flush()
     return num ** 2

q = Queue()
pool = Pool(3, initializer, [q])

for _ in pool.map(task, range(5)):
    proc, out = q.get()
    print proc, "got", out

This should print something like this:

PoolWorker-1 got 0
PoolWorker-1 got 3
PoolWorker-1 got 4
PoolWorker-2 got 1
PoolWorker-3 got 2

Don't forget to call sys.{stdout,stderr}.flush() at the end of task otherwise, nothing will be written to the queue.