Parfor for Python

Question

I am looking for a definitive answer to MATLAB\'s parfor for Python (Scipy, Numpy).

Is there a solution similar to parfor? If not, what is the complication for creat

Answer 1

I tried all solutions here, but found that the simplest way and closest equivalent to matlabs parfor is numba's prange.

Essentially you change a single letter in your loop, range to prange:

from numba import autojit, prange

@autojit
def parallel_sum(A):
    sum = 0.0
    for i in prange(A.shape[0]):
        sum += A[i]

    return sum

Answer 2

I recommend trying joblib Parallel.

one liner

from joblib import Parallel, delayed
out = Parallel(n_jobs=2)(delayed(heavymethod)(i) for i in range(10)) 

---

instructional

instead of taking a for loop

from time import sleep
for _ in range(10):
   sleep(.2)

rewrite your operation into a list comprehension

[sleep(.2) for _ in range(10)]

Now let us not directly evaluate the expression, but collect what should be done. This is what the delayed method is for.

from joblib import delayed
[delayed(sleep(.2)) for _ in range(10)]

Next instantiate a parallel process with n_workers and process the list.

from joblib import Parallel
r = Parallel(n_jobs=2, verbose=10)(delayed(sleep)(.2) for _ in range(10)) 

[Parallel(n_jobs=2)]: Done   1 tasks      | elapsed:    0.6s
[Parallel(n_jobs=2)]: Done   4 tasks      | elapsed:    0.8s
[Parallel(n_jobs=2)]: Done  10 out of  10 | elapsed:    1.4s finished