How can l launch each worker in a multiprocessing.Pool in a new shell?

Question

I'm trying to couple the execution of a spawned process in a worker pool to a new system terminal. In the following example (adapted from @sylvain-leroux's answer to this question) a pool of workers is constructed to do some work with queued objects.

import os
import time
import multiprocessing

# A main function, to be run by our workers.
def worker_main(queue):
    print('The worker at', os.getpid(), 'is initialized.')
    while True:

        # Block until something is in the queue.
        item = queue.get(True)
        print(item)
        time.sleep(0.5)

if __name__ == '__main__':

    # Instantiate a Queue for communication.
    the_queue = multiprocessing.Queue()

    # Build a Pool of workers, each running worker_main.
    the_pool = multiprocessing.Pool(3, worker_main, (the_queue,))

    # Iterate, sending data via the Queue.
    for i in range(5):
        the_queue.put("That's a nice string you got there.")
        the_queue.put("It'd be a shame if something were to... garble it.")

    worker_pool.close()
    worker_pool.join()
    time.sleep(10)

If you run this from a system terminal you'll see a bunch of garbled text, because each of the workers is writing out to, and executing in, the same console. For an project I'm working on, it would be helpful to spawn a new shell/console to host each worker process, such that all printed output is displayed in that shell, and the execution of the worker process is host in that shell. I've seen several examples doing this with Popen using the shell keyword, but I need to stick to a pool-based implementation, due to compatibility constraints. Has anyone out there done this? Guidance is appreciated.

Answer 1

Try using the Queue the other way around.

Let the workers put messages into the Queue, and in the parent process get them from the Queue and print them. That should get rid of intermingled output.

If you want to pass messages both from parent to workers and back, use two queues. One for passing messages to workers, and one to pass messages back to the parent.