Pandas GroupBy memory deallocation

Question

Problem

I noticed that memory allocated while iterating through a Pandas GroupBy object is not deallocated after iteration. I use resource.getrusage(resource.RUSA

Answer 1

Why is memory for DataFrames resulting from iteration through the groupby not deallocated after iteration is completed?

Nowhere in your code you del object gb, which means at the end it's still there. One thing is to have an iterator reach the end of its cycle, then I would expect it to die automagically, but the object that gave rise to the iterator persists, in case you need to do something else (iterate again, aggregate, etc).

Answer 2

Memory Weirdness

This is very interesting! You do not need del idx, x. Only using gc.collect() worked to keep memory constant for me. This is much cleaner that having the del statements inside the loop.