Python subprocess.Popen erroring with OSError: [Errno 12] Cannot allocate memory after period of time

Question

Note: This question has been re-asked with a summary of all debugging attempts here.

---

I have a Python script that is running as a background pr

Answer 1

Virtual Memory matters!!!

I encountered the same issue before I add swap to my OS. The formula for virtual memory is usually like: SwapSize + 50% * PhysicalMemorySize. I finally get this resolved by either adding more physical memory or adding a Swap disk. close_fds won't work in my case.

Answer 2

You need to

ps = subprocess.Popen(["sleep", "1000"])
os.waitpid(ps.pid, 0)

to free resources.

Note: this does not work on Windows.

Answer 3

That swap space answer is bogus. Historically Unix systems wanted swap space available like that, but they don't work that way anymore (and Linux never worked that way). You're not even close to running out of memory, so that's not likely the actual problem - you're running out of some other limited resource.

Given where the error is occuring (_get_handles calls os.pipe() to create pipes to the child), the only real problem you could be running into is not enough free file descriptors. I would instead look for unclosed files (lsof -p on the PID of the process doing the popen). If your program really needs to keep a lot of files open at one time, then increase the user limit and/or the system limit for open file descriptors.