Parallel execution of a list of functions

Question

So using the multiprocess module it is easy to run a function in parallel with different arguments like this:

from multiprocessing import Pool

def f(x):
    ret         


        

Answer 1

You can get a tuple returned. This could be done quite easily and in a very compact way using the lightweight module: joblib. I recommend joblib because it is lightweight

from joblib import Parallel, delayed
import multiprocessing
import timeit


# Implementation 1
def f(x):    
    return x**2, x**3 + 2


#Implementation 2 for a more sophisticated second or more functions
def g(x):
    return  x**3 + 2


def f(x):    
    return x**2, g(x)


if __name__ == "__main__":
  inputs = [i for i in range(32)]
  num_cores = multiprocessing.cpu_count()
  t1 = timeit.Timer()
  result = Parallel(n_jobs=num_cores)(delayed(f)(i) for i in inputs)
  print(t1.timeit(1)) 

Using multiprocessing.Pool as you already have in the question

from multiprocessing import Pool, cpu_count
import timeit


def g(x):
    return x**3 + 2


def f(x):    
    return x**2, g(x)

if __name__ == "__main__":
  inputs = [i for i in range(32)]
  num_cores = cpu_count()
  p = Pool(num_cores)
  t1 = timeit.Timer()
  result = p.map(f, inputs)
  print(t1.timeit(1))
  print(result)    

Example Output:

print(result)

[(0, 2), (1, 3), (4, 10), (9, 29), (16, 66), (25, 127), (36, 218), (49, 345), 
(64, 514), (81, 731), (100, 1002), (121, 1333), (144, 1730), (169, 2199), 
(196, 2746), (225, 3377), (256, 4098), (289, 4915), (324, 5834), (361, 6861), 
(400, 8002), (441, 9263), (484, 10650), (529, 12169), (576, 13826), (625, 
15627), (676, 17578), (729, 19685), (784, 21954), (841, 24391), (900, 27002), 
(961, 29793)]

print(t1.timeit(1))
    5.000001692678779e-07         #(with 16 cpus and 64 Gb RAM) 

for: inputs = range(2000), it took the time: 1.100000190490391e-06