Reducing numpy memory footprint in long-running application

Question

In my application one hundred numpy arrays (1000 complex elements each) are generated and filled with data. Then over many iterations, the array elements are modified over and o

Answer 1

Via using the system monitor and code inspection/commenting, I found the memory leak. It was caused by comparing a numpy array with an empty list in a different file. I will dig into the leak in a different place and vote to delete this question as I find it too specific to assist anyone else.

[Update 1]: New question describing the source of the problem: Why does comparison of a numpy array with a list consume so much memory?

Answer 2

I had the same kind of issue. Unfortunately I found no workaround. The only thing that worked for me was to refactor the code into small isolated function. Those functions should be made so that I can convince myself I keep no references to arrays that keep the garbage collector from collecting arrays. You should take special care to array views which are generated by slicing, etc ...

In order to make code more memory efficient and less prone to memory leaks, I often found usefull to use the out= keyword argument provided by many numpy functions.