Python memory footprint vs. heap size

Question

I\'m having some memory issues while using a python script to issue a large solr query. I\'m using the solrpy library to interface with th

Answer 1

CPython implementation only exceptionally free's allocated memory. This is a widely known bug, but it isn't receiving much attention by CPython developers. The recommended workaround is to "fork and die" the process that consumes lots RAM.

Answer 2

What version of python are you using?
I am asking because older version of CPython did not release the memory and this was fixed in Python 2.5.

Answer 3

I've implemented hruske's advice of "fork and die". I'm using os.fork() to execute the memory intensive section of code in a child process, then I let the child process exit. The parent process executes an os.waitpid() on the child so that only one thread is executing at a given time.

If anyone sees any pitfalls with this solution, please chime in.

Answer 4

Python allocates Unicode objects from the C heap. So when you allocate many of them (along with other malloc blocks), then release most of them except for the very last one, C malloc will not return any memory to the operating system, as the C heap will only shrink on the end (not in the middle). Releasing the last Unicode object will release the block at the end of the C heap, which then allows malloc to return it all to the system.

On top of these problems, Python also maintains a pool of freed unicode objects, for faster allocation. So when the last Unicode object is freed, it isn't returned to malloc right away, making all the other pages stuck.