Does functools.partial not work with multiprocessing.Pool.map?

问题

I have code that, simplified down, looks like this:

run = functools.partial(run, grep=options.grep, print_only=options.print_only, force=options.force)

if not options.single and not options.print_only and options.n > 0:
    pool = multiprocessing.Pool(options.n)
    Map = pool.map
else: Map = map

for f in args:
    with open(f) as fh: Map(run, fh)

try:
    pool.close()
    pool.join()
except NameError: pass

That works fine when I run it in single process mode, but fails with errors like this

TypeError: type 'partial' takes at least one argument

mixed up together with long call stacks through the multiprocessing module. What's going on?

I'm using python 2.6.1.

回答1:

Google tells me that this is a bug in Python; apparently fixed in Py3k. It's supposedly due to partial not being picklable.

There is a workaround.

回答2:

Its works using:

from multiprocessing.dummy import Pool as ThreadPool

def executeFunction(server,function):
    print "Server: %s and function: %s" % (server,function)


def executeParallel(servers,function, threads=10):
    pool = ThreadPool(threads)
    functionArguments=partial(executeFunction, function=function)
    results = pool.map(functionArguments, servers)
    pool.close()
    pool.join()
    return results

executeParallel(serverList, "Install")

来源：https://stackoverflow.com/questions/3637847/does-functools-partial-not-work-with-multiprocessing-pool-map