Python manager.dict() is very slow compared to regular dict

Question

I have a dict to store objects:

jobs = {}
job = Job()
jobs[job.name] = job

now I want to convert it to use manager dict because I want to use multiprocessing and need to share this dict amonst processes

mgr = multiprocessing.Manager()
jobs = mgr.dict()
job = Job()
jobs[job.name] = job

just by converting to use manager.dict() things got extremely slow.

For example, if using native dict, it only took .65 seconds to create 625 objects and store it into the dict.

The very same task now takes 126 seconds!

Any optimization i can do to keep manager.dict() on par with python {}?

Answer 1

The problem is that each insert is quite slow for some reason (117x slower on my machine), but if you update your manager.dict() with a normal dict, it will be a single and fast operation.

jobs = {}
job = Job()
jobs[job.name] = job
# insert other jobs in the normal dictionary

mgr = multiprocessing.Manager()
mgr_jobs = mgr.dict()
mgr_jobs.update(jobs)

Then use the mgr_jobs variable.

Another option is to use the widely adopted multiprocessing.Queue class.

Answer 2

If you are using mgr.dict() inside a loop in your pool. You can use a local normal dict to store results temporarily and then update your mgr.dict() outside the loop like your_mgr_dict.update(local_dict)