Python multiprocessing with an updating queue and an output queue

后端 未结 2 1655
青春惊慌失措
青春惊慌失措 2021-02-04 17:32

How can I script a Python multiprocess that uses two Queues as these ones?:

  1. one as a working queue that starts with some data and that, depending on conditions of
2条回答
  •  一个人的身影
    2021-02-04 18:22

    The following code achieves the expected results. It follows the suggestions made by @tawmas.

    This code allows to use multiple cores in a process that requires that the queue which feeds data to the workers can be updated by them during the processing:

    import multiprocessing as mp
    def worker(working_queue, output_queue):
        while True:
            if working_queue.empty() == True:
                break #this is the so-called 'poison pill'    
            else:
                picked = working_queue.get()
                if picked % 2 == 0: 
                        output_queue.put(picked)
                else:
                    working_queue.put(picked+1)
        return
    
    if __name__ == '__main__':
        static_input = xrange(100)    
        working_q = mp.Queue()
        output_q = mp.Queue()
        results_bank = []
        for i in static_input:
            working_q.put(i)
        processes = [mp.Process(target=worker,args=(working_q, output_q)) for i in range(mp.cpu_count())]
        for proc in processes:
            proc.start()
        for proc in processes:
            proc.join()
        results_bank = []
        while True:
           if output_q.empty() == True:
               break
           results_bank.append(output_q.get_nowait())
        print len(results_bank) # length of this list should be equal to static_input, which is the range used to populate the input queue. In other words, this tells whether all the items placed for processing were actually processed.
        results_bank.sort()
        print results_bank
    

提交回复
热议问题