Python multiprocessing doesn't seem to use more than one core

Question

I want to use Python multiprocessing to run grid search for a predictive model. When I look at core usage, it always seem to be using only one core. Any idea what I\'m doing wro

Answer 1

According to the documentation the join() command locks the current thread until the specified thread returns. So you are basically starting each thread in the for loop and then wait for it to finish, BEFORE you proceed to the next iteration.

I would suggest moving the joins outside the loop!

Answer 2

I'd say :

for g in grid:
    g.p = multiprocessing.Process(target=worker, args=(g,GRID_hx))
    jobs.append(g.p)
    g.p.start()
for g in grid:
    g.p.join()

Currently you're spawning a job, then waithing for it to be done, then going to the next one.

Answer 3

Your problem is that you join each job immediately after you started it:

for g in grid:
    p = multiprocessing.Process(target=worker, args=(g,GRID_hx))
    jobs.append(p)
    p.start()
    p.join()

join blocks until the respective process has finished working. This means that your code starts only one process at once, waits until it is finished and then starts the next one.

In order for all processes to run in parallel, you need to first start them all and then join them all:

jobs = []
for g in grid:
    p = multiprocessing.Process(target=worker, args=(g,GRID_hx))
    jobs.append(p)
    p.start()

for j in jobs:
    j.join()

Documentation: link