Multiprocessing in a pipeline done right

Question

I\'d like to know how multiprocessing is done right. Assuming I have a list [1,2,3,4,5] generated by function f1 which is written to a Queue

Answer 1

The easiest way to do exactly that is using semaphores.

F1

F1 is populating your 'Queue' with the data you want to process. End the end of this push, you put n 'Stop' keywords in your queue. n = 2 for your example, but usually the number of involved workers. Code would look like:

for n in no_of_processes:
    tasks.put('Stop')

F2

F2 is pulling from the provided queue by a get-command. The element is taken from the queue and deleted in the queue. Now, you can put the pop into a loop while paying attention to the stop signal:

for elem in iter(tasks.get, 'STOP'):
   do something

F3

This one is a bit tricky. You could generate a semaphore in F2 that acts as a signal to F3. But you do not know when this signal arrives and you may loose data. However, F3 pulls the data the same way as F2 and you could put that into a try... except-statement. queue.get raises an queue.Emptywhen there are no elements in the queue. So your pull in F3 would look like:

while control:
    try:
        results.get()
    except queue.Empty:
        control = False

With tasks and results being queues. So you do not need anything which is not already included in Python.