Getting “Queue objects should only be shared between processes through inheritance” but I'm not using a Queue

问题

I am trying to use a ProcessPoolExecutor, but I am getting the error "Queue objects should only be shared between processes through inheritance", but I am not using a Queue (at least not explicitly). I can't find anything that explains what I am doing wrong.

Here is some code that demonstrates the issue (not my actual code):

from concurrent.futures import ProcessPoolExecutor, as_completed

class WhyDoesntThisWork:

    def __init__(self):
        self.executor = ProcessPoolExecutor(4)

    def execute_something(self, starting_letter):
        futures = [self.executor.submit(self.something, starting_letter, d) for d in range(4)]
        letter = None
        for future in as_completed(futures):
            letter = future.result()
        print(letter)

    def something(self, letter, d):
        # do something pointless for the example
        for x in range(d):
            letter = chr(ord(letter) + 1)

if __name__ == '__main__':
    WhyDoesntThisWork(). execute_something('A')

El Ruso has pointed out that making something() a staticmethod or classmethod makes the error go away. Unfortunately, my actual code needs to call other methods using self.

回答1:

try this code for something

@staticmethod
def something(letter, d):
    # do something pointless for the example
    for x in range(d):
        letter = chr(ord(letter) + 1)

or refactor to:

from concurrent.futures import ProcessPoolExecutor, as_completed


class WhyDoesntThisWork:
    def something(self, letter, d):
        # do something pointless for the example
        for x in range(d):
            letter = chr(ord(letter) + 1)
        return letter


if __name__ == '__main__':
    executor = ProcessPoolExecutor(4)
    letter = 'A'
    obj = WhyDoesntThisWork()
    futures = [executor.submit(obj.something, letter, d) for d in range(4)]
    for future in as_completed(futures):
        print(future.result())

回答2:

Can be solved without using static approach.

When using process, each process runs in an independent memory space. That's unlike when using thread, when different threads are running under the same process, using the same memory space. Thus the error doesn't occur when you use ThreadPoolExecutor but occurs in ProcessPoolExecutor.

So when the function of the class instance is delivered into separate sub-processes, the multiprocessing mechanism pickles the function so that the function can be passed into the sub-process as an independent instance. And when the sub-process joined, the class is updated by the function instance delivered back as unpicked one.

To make it work, just add __getstate__() and __setstate__() functions to the class to guide the class how to pickle and unpickle the function. In pickling, the unnecessary fields can be excluded as shown in del self_dict['executor'].

import multiprocessing
import time
from concurrent.futures import ProcessPoolExecutor, as_completed


class GuessItWorksNow():

    def __init__(self):
        self.executor = ProcessPoolExecutor(4)

    def __getstate__(self):
        state = self.__dict__.copy()
        del state['executor']
        return state

    def __setstate__(self, state):
        self.__dict__.update(state)

    def something(self, letter, d):
        # do something pointless for the example
        p = multiprocessing.current_process()
        time.sleep(1)
        for x in range(d):
            letter = chr(ord(letter) + 1)
        return (f'[{p.pid}] ({p.name}) ({letter})')

    def execute_something(self, starting_letter):
        futures = [self.executor.submit(self.something, starting_letter, d) for d in range(10)]
        for future in as_completed(futures):
            print(future.result())


if __name__ == '__main__':
    obj = GuessItWorksNow()
    obj.execute_something('A')

来源：https://stackoverflow.com/questions/47163820/getting-queue-objects-should-only-be-shared-between-processes-through-inheritan

标签

python

python-multiprocessing

concurrent.futures