Sharing state between forked worker processes in a high-performance environment

Question

This is a follow up to my previous question. As suggested by Tim Peters, using a Manager may not necessarily be the best approach. Unfortunately I\'ve got too m

Answer 1

The scenario you describe, you are likely to have large performance issues due to the GIL when using multi-threading. Probably to avoid that you chose to use multi processing instead. That, on the other hand, uses processes, so data structures might get copied for each subprocess.

I hate to say it, but using a non-Python solution (e. g. in C++) might speed up things because there you do not have the GIL problem. Then you can use multi-threading, do not have to copy things etc. Reading from a large dictionary from several threads is not really an issue, so you won't have to synchronize anything (what the GIL always would do for you without a real need).