Generator function of child processes runs in the Parent process

Question

I am trying to run a generator process in parallel by child processes. But when I tried to do this, I see the function with generator was processed by the parent process!!!

Answer 1

One way of achieving communications between two processes is by using a Queue instance. In the following example, instead of creating two individual processes I have opted instead to create a process pool of two processes:

from multiprocessing import Pool, Manager
import os

def p(q):
    pid = os.getpid()
    q.put(pid)
    for i in range(5):
        q.put(i)
    q.put(None) # signify "end of file"


def main():
    manager = Manager()
    q1 = manager.Queue()
    q2 = manager.Queue()
    with Pool(2) as pool: # create a pool of 2 processes
        pool.apply_async(p, args=(q1,))
        pool.apply_async(p, args=(q2,))
        q1_eof = False
        q2_eof = False
        while not q1_eof or not q2_eof:
            if not q1_eof:
                obj = q1.get() # blocking get
                if obj is None:
                    q1_eof = True
                else:
                    print(obj)
            if not q2_eof:
                obj = q2.get() # blocking get
                if obj is None:
                    q2_eof = True
                else:
                    print(obj)

if __name__ == '__main__':
    main()

Prints:

5588
24104
0
0
1
1
2
2
3
3
4
4

The code that uses explicit Process instances rather than creating a pool follows (I don't tend to subclass the Process class as that requires more coding:)

from multiprocessing import Process, Queue
import os

def p(q):
    pid = os.getpid()
    q.put(pid)
    for i in range(5):
        q.put(i)
    q.put(None) # signify "end of file"


def main():
    q1 = Queue()
    q2 = Queue()
    p1 = Process(target=p, args=(q1,))
    p1.start()
    p2 = Process(target=p, args=(q2,))
    p2.start()
    q1_eof = False
    q2_eof = False
    while not q1_eof or not q2_eof:
        if not q1_eof:
            obj = q1.get() # blocking get
            if obj is None:
                q1_eof = True
            else:
                print(obj)
        if not q2_eof:
            obj = q2.get() # blocking get
            if obj is None:
                q2_eof = True
            else:
                print(obj)
    p1.join()
    p2.join()

if __name__ == '__main__':
    main()

Important Note

The two coding examples (one that uses a process pool and one that doesn't) uses two different types of Queue instances.

See Python multiprocessing.Queue vs multiprocessing.manager().Queue()

You can always use multiprocessing.manager().Queue() in all cases (I generally do) but at a possible loss of some efficiency.