Python 3.5 multiprocessing pool and queue don't work

问题

I encounter a multiprocessing problem. The code is included below. The code can execute as expected, but when uncommenting self.queue = multiprocessing.Queue(), this program will exit immediately and it seems that the subprocess can't be started successfully.

I don't know what happened. Could someone help me out? Many Thanks!

import multiprocessing
import time

class Test:
    def __init__(self):
        self.pool = multiprocessing.Pool(1)
        #self.queue = multiprocessing.Queue()

    def subprocess(self):
        for i in range(10):
            print("Running")
            time.sleep(1)
        print("Subprocess Completed")

    def start(self):
        self.pool.apply_async(func=self.subprocess)
        print("Subprocess has been started")
        self.pool.close()
        self.pool.join()

    def __getstate__(self):
        self_dict = self.__dict__.copy()
        del self_dict['pool']
        return self_dict

    def __setstate__(self, state):
        self.__dict__.update(state)

if __name__ == '__main__':
    test = Test()
    test.start()

回答1:

I can reproduce your Issue and also no Traceback raised up.
This should raise the following error, don't know why not:

RuntimeError: Queue objects should only be shared between processes through inheritance

Replace your line of code with:

    m = multiprocessing.Manager()
    self.queue = m.Queue()

Why does this happens?
multiprocessing.Queue is for use with one Process, you are using multiprocessing.Pool, which uses multiple Process, you have to use multiprocessing.Manager().Queue().

Tested with Python: 3.4.2

回答2:

you use apply_async, which returns immediately. so you should wait for the result somewhere
under the hood, python will pickle the function to be executed to the child process. but self.process as a method is not pickle-able here (because of the self.pool attribute, see comment by ShadowRanger below).

import multiprocessing
import time

def subprocess():   # is a plain old (pickle-able) function
    for i in range(10):
        print("Running")
        time.sleep(1)
    print("Subprocess Completed")
    return True

class Test:
    def __init__(self):
        self.pool = multiprocessing.Pool(1)

    def start(self):
        result = self.pool.apply_async(subprocess)
        print("Subprocess has been started")
        result.get()    # wait for the end of subprocess
        self.pool.close()
        self.pool.join()

if __name__ == '__main__':
    test = Test()
    test.start()

来源：https://stackoverflow.com/questions/43155553/python-3-5-multiprocessing-pool-and-queue-dont-work

标签

python

python-3.x

multiprocessing