Pool within a Class in Python

Question

I would like to use Pool within a class, but there seems to be a problem. My code is long, I created a small-demo variant to illustrated the problem. It would be great if yo

Answer 1

It looks like because of the way the function gets passed to the worker threads (pickling) you can't use instance methods unfortunately. My first thought was to use lambdas, but it turns out the built in pickler can't serialize those either. The solution, sadly, is just to use a function in the global namespace. You can still make it an instance attribute though, take a look:

from multiprocessing import Pool

def F(x):
    return x * x

class SeriesInstance(object):
    def __init__(self):
        self.numbers = [1,2,3]
        self.F = F

    def run(self):
        p = Pool()
        out = p.map(self.F, self.numbers)
        p.close()
        p.join()
        return out

if __name__ == '__main__':
    print SeriesInstance().run()

Answer 2

You can also use multiprocessing with static functions in the class.

Answer 3

You have an error, because pickle can't serialize instancemethod. So you should use this tiny workaround:

from itertools import repeat
from multiprocessing import Pool


class SeriesInstance:
    def __init__(self):
        self.numbers = [1, 2, 3]

    def F(self, x):
        return x * x

    def run(self):
        p = Pool()
        print(list(p.starmap(SeriesInstance.F, zip(repeat(self), self.numbers))))


if __name__ == '__main__':
    SeriesInstance().run()