Python: Memory usage and optimization when modifying lists

Question

The problem

My concern is the following: I am storing a relativity large dataset in a classical python list and in order to process the data I must iterate over the li

Answer 1

A set (or even a dict) might be what you're looking for. It's the same underlying structure as a dictionary (without the associated values), but your objects do need to be hashable.

If order is important in your list/set, you could make an ordered set. There is a good recipe on activestate for an OrderedSet. There is another slick suggestion in this answer. Python 2.7 and 3.1 also have an an OrderedDict You would ave to test the implementation for your self to see how the overhead impacts you, but the speed gains from the hashtable may well be worth it.

Depending on what sort of comparisons you're making on the objects in the list, a heap (heapq module) might also fit your problem. The heap will minimize the number of operations required for inserting and removing items in the underlying list.