function initialize and object initialization (multiprocessing)

问题

I recently saw a answer/comment about how functions are objects in python. So, I wondering why when I take that example, and create a class around it while initializing a variable, it doesn't work the same way. (The class example receives a pickling error): PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

Does anyone know why this is?

Example code from the link:

import multiprocessing as mp

def f(x):
    f.q.put('Doing: ' + str(x))
    return x*x

def f_init(q):
    f.q = q

def main():
    jobs = range(1,6)

    q = mp.Queue()
    p = mp.Pool(None, f_init, [q])
    results = p.imap(f, jobs)
    p.close()

    for i in range(len(jobs)):
        print q.get()
        print results.next()

if __name__ == '__main__':
    main()

Same example while puttin f into a class:

import multiprocessing as mp

class F:
    def __init__(self, q):
        self.q = q
    def f(x):
        self.q.put('Doing: ' + str(x))
        return x*x

def main():
    jobs = range(1,6)

    q = mp.Queue()
    p = mp.Pool(None)
    f = F(q)
    results = p.imap(f.f, jobs)
    p.close()

    for i in range(len(jobs)):
        print q.get()
        print results.next()

if __name__ == '__main__':
    main()

回答1:

Instance methods are not automatically picklable. So

p.imap(f.f, jobs)

fails because p.imap tries to pickle the arguments. There is a way to "teach" pickle how to pickle instance methods (see Steven Bethard's answer), but your code has another problem: Passing the queue to the instance leads to a RuntimeError:

RuntimeError: Queue objects should only be shared between processes through inheritance

The error message is a little confusing (at least to me) since you can pass the queue as an argument to p.imap, but you can not pass it to the class F first, and then transfer it to the worker processes through f.f.

Anyway, because of these problems, I'd suggest sticking with the original code instead of trying to wrap the code in a class.

Here is an example of how to pickle instance methods:

import multiprocessing as mp
import copy_reg
import types

def _pickle_method(method):
    # Author: Steven Bethard (author of argparse)
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-
    # instancemethods
    func_name = method.im_func.__name__
    obj = method.im_self
    cls = method.im_class
    cls_name = ''
    if func_name.startswith('__') and not func_name.endswith('__'):
        cls_name = cls.__name__.lstrip('_')
    if cls_name:
        func_name = '_' + cls_name + func_name
    return _unpickle_method, (func_name, obj, cls)


def _unpickle_method(func_name, obj, cls):
    # Author: Steven Bethard
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-
    # instancemethods
    for cls in cls.mro():
        try:
            func = cls.__dict__[func_name]
        except KeyError:
            pass
        else:
            break
    return func.__get__(obj, cls)

# This call to copy_reg.pickle allows you to pass methods as the first arg to
# mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in
# PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
# __builtin__.instancemethod failed

copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method)

class F(object):
    def f(self, x):
        fq.put('Doing: ' + str(x))        
        return x*x

def f_init(q):
    # http://stackoverflow.com/a/3843313/190597 (Olson)
    global fq
    fq = q

def main():
    jobs = range(1,6)
    q = mp.Queue()
    p = mp.Pool(None, f_init, [q]) 
    f = F()
    results = p.imap(f.f, jobs)
    p.close()

    for r in results:
        print(r, q.get())

if __name__ == '__main__':
    main()

yields

(1, 'Doing: 2')
(4, 'Doing: 3')
(9, 'Doing: 4')
(16, 'Doing: 1')
(25, 'Doing: 5')

来源：https://stackoverflow.com/questions/15932565/function-initialize-and-object-initialization-multiprocessing

标签

python

oop

multiprocessing