How to use multiprocessing pool.map with multiple arguments?

Question

In the Python multiprocessing library, is there a variant of pool.map which supports multiple arguments?

text = "test"
def         


        

Answer 1

A better way is using decorator instead of writing wrapper function by hand. Especially when you have a lot of functions to map, decorator will save your time by avoiding writing wrapper for every function. Usually a decorated function is not picklable, however we may use functools to get around it. More disscusions can be found here.

Here the example

def unpack_args(func):
    from functools import wraps
    @wraps(func)
    def wrapper(args):
        if isinstance(args, dict):
            return func(**args)
        else:
            return func(*args)
    return wrapper

@unpack_args
def func(x, y):
    return x + y

Then you may map it with zipped arguments

np, xlist, ylist = 2, range(10), range(10)
pool = Pool(np)
res = pool.map(func, zip(xlist, ylist))
pool.close()
pool.join()

Of course, you may always use Pool.starmap in Python 3 (>=3.3) as mentioned in other answers.