问题
I have a database composed of millions of training examples. Each is saved as its own numpy.memmap. (Yes, yes, I know, but they're of irregular sizes. I probably will modify my design to put like-size examples together in one memmap and hide that fact from the user.)
Trying to open this database causes me to run in to the system NOFILES limits, but I've solved that part.
Now I'm running in to OSError: [Errno 12] Cannot allocate memory after about 64865 memmaps are created, and executing most other code after that point results in a MemoryError. This is strange because the process takes only 1.1 GiB of memory before it fails, and this server has nearly 100 GiB of memory.
I've gone and saved a million dummy files in a directory and opened them all with python's standard open function, and it works fine. Takes about 5 GiB of memory between file handles and contents, but it works.
What could be limiting me to only opening about 2^16 memory maps?
来源：https://stackoverflow.com/questions/57402397/opening-millions-of-numpy-memmaps-in-python