Python: Memory usage and optimization when modifying lists

Question

The problem

My concern is the following: I am storing a relativity large dataset in a classical python list and in order to process the data I must iterate over the li

Answer 1

Brandon Craig Rhodes suggests using a collections.deque, which can suit this problem: no additional memory is required for the operation and it is kept O(n). I do not know the total memory usage and how it compares to a list; it's worth noting that a deque has to store a lot more references and I would not be surprised if it isn't as memory intensive as using two lists. You would have to test or study it to know yourself.

If you were to use a deque, I would deploy it slightly differently than Rhodes suggests:

from collections import deque
d = deque(range(30))
n = deque()

print d

while True:
    try:
        item = d.popleft()
    except IndexError:
        break

    if item % 3 != 0:
        n.append(item)

print n

There is no significant memory difference doing it this way, but there's a lot less opportunity to flub up than mutating the same deque as you go.