Python garbage collection can be that slow?

Question

I have a problem with my python application, and I think it\'s related to the python garbage collection, even if I\'m not sure...

The problem is that my application

Answer 1

This is known garbage collector issue in Python 2.6 causing quadratic time for garbage collection when many objects are being allocated without deallocating any of them ie. population of large list.
There are two simple solutions:

1. either disable garbage collection before populating large lists and enable it afterwards

   l = []
   gc.disable()
   for x in xrange(10**6):
     l.append(x)
   gc.enable()
   

2. or update to Python 2.7, where the issue has been solved

I prefer the second solution, but it's not always an option;)

Answer 2

If your problem really is the garbage collection, try explicitly freeing your objects when you're done with them using del().

In general, this doesn't sound like a garbage collection problem, unless we're talking about terabytes of memory.

I agree with S.Lott... profile your app, then bring code snippets and the results of that back and we can be much more helpful.

Answer 3

Yes, it could be garbage collection, but it could also be some synchronisation with the C++ code, or something completely different (hard to say without code).

Anyway, you should have a look at SIG for development of Python/C++ integration to find issues and how to speed up things.