Multiprocessing - using the Managers Namespace to save memory

Question

I have several processes each completing tasks which require a single large numpy array, this is only being read (the threads are searching it for appropriate values).

Answer 1

Manager types are built for flexibility not efficiency. They create a server process that holds the values, and can return proxy objects to each process they are needed in. The server and proxy communicate over tls to allow the server and proxy to be on different machines, but this necessarily means copying whatever object is in question. I haven't traced the source all the way, so it's possible the extra copy may be garbage collected after use, but at least initially there has to be a copy.

If you want shared physical memory, I suggest using Shared ctypes Objects. These actually do point to a common location in memory, and therefore are much faster, and resource-light. They do not support all the same things full fat python objects do, but they can be extended by creating structs to organize your data.