Child to parent communication in Python multiprocessing

Question

I am writing a python script that will parse through a file quickly by sending lines to different processes to handle. At the end, I want the parent to receive the results f

Answer 1

I guess the issue is that you are trying to access object copy of which was actually changed in another process (child). You need to use explicit inter-process communication to send response back to parent. I.e. something like that JoinableQueue you use to pass information to children.

btw, I can't see how you send your data to consumers.

Answer 2

You can pass values back through an results queue.

In LineConsumer:

def __init__(self, queue, result_queue):
   self.result_queue = result_queue
   # ...

def terminate(self):
   self.results_queue.put(self.lines)
   super(LineConsumer, self).terminate()

In Parser:

queue = JoinableQueue(100)
result_queue = Queue()
# ...
  lc = LineConsumer(queue, result_queue)
# ...
for p in consumers:
  p.terminate()
  p.join()

while True:
  try:
    print results.queue.get(False)
  except Queue.Empty: # need to import Queue
    break