Python multiprocessing with an updating queue and an output queue

Question

How can I script a Python multiprocess that uses two Queues as these ones?:

1. one as a working queue that starts with some data and that, depending on conditions of

Answer 1

You have a typo in the line that creates the processes. It should be mp.Process, not mp.process. This is what is causing the exception you get.

Also, you are not looping in your workers, so they actually only consume a single item each from the queue and then exit. Without knowing more about the required logic, it's not easy to give specific advice, but you will probably want to enclose the body of your worker function inside a while True loop and add a condition in the body to exit when the work is done.

Please note that, if you do not add a condition to explicitly exit from the loop, your workers will simply stall forever when the queue is empty. You might consider using the so-called poison pill technique to signal the workers they may exit. You will find an example and some useful discussion in the PyMOTW article on Communication Between processes.

As for the number of processes to use, you will need to benchmark a bit to find what works for you, but, in general, one process per core is a good starting point when your workload is CPU bound. If your workload is IO bound, you might have better results with a higher number of workers.