Python manager.dict() is very slow compared to regular dict

Question

I have a dict to store objects:

jobs = {}
job = Job()
jobs[job.name] = job

now I want to convert it to use manager dict because I want to use m

Answer 1

The problem is that each insert is quite slow for some reason (117x slower on my machine), but if you update your manager.dict() with a normal dict, it will be a single and fast operation.

jobs = {}
job = Job()
jobs[job.name] = job
# insert other jobs in the normal dictionary

mgr = multiprocessing.Manager()
mgr_jobs = mgr.dict()
mgr_jobs.update(jobs)

Then use the mgr_jobs variable.

Another option is to use the widely adopted multiprocessing.Queue class.

Answer 2

If you are using mgr.dict() inside a loop in your pool. You can use a local normal dict to store results temporarily and then update your mgr.dict() outside the loop like your_mgr_dict.update(local_dict)