All but the last N elements of iterator in Python

Question

What is the best way to get all but the last N elements of an iterator in Python? Here is an example of it in theoretical action:

>>> list(all_but_         


        

Answer 1

Use a collections.deque. Push N items from the source on the first invocation. On each subsequent invocation, pop an item out, push an item in from the source, and yield the popped item.

Answer 2

Based on Ignacio Vazquez-Abrams's description:

from collections import deque

def all_but_the_last_n(iterable, count):
    q = deque()
    i = iter(iterable)
    for n in range(count):
        q.append(i.next())
    for item in i:
        q.append(item)
        yield q.popleft()

I wondered whether it was better to use the deque right to left (append, popleft) or left to right (appendleft, pop). So i timed it with python 2.5.2 and found that rtl was 3.59 usec while ltr was 3.53 usec. The difference of 0.06 usec is not significant. the test was to append a single item and pop a single item.

Answer 3

For a list you could do this:

def all_but_the_last_n(aList, N):
    return aList[:len(aList) - N]

myList = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
N = 4

print(all_but_the_last_n(myList, N))

Will print:

[0, 1, 2, 3, 4, 5]

Answer 4

Just for the fun of it, here's a variation on Ignacio's solution that doesn't require a deque.

>>> def truncate(it, n):
...     cache = [next(it) for i in range(n)]
...     index = 0
...     for val in it:
...         val, cache[index] = cache[index], val
...         index = (index + 1) % n
...         yield val

I wasn't especially concerned with speed when I wrote the above... but perhaps this would be a tad faster:

def truncate(it, n):
    cache = [next(it) for i in range(n)]
    index = 0
    for val in it:
        yield cache[index]
        cache[index] = val
        index = (index + 1) % n

Answer 5

This is compact (aside from consume recipe taken from itertools recipes) and should run at C speed:

import collections
import operator
from itertools import islice, tee


def truncate(iterable, n):
    a, b = tee(iterable)
    consume(b, n)
    return map(operator.itemgetter(0), zip(a, b))


# From itertools recipes
def consume(iterator, n=None):
    "Advance the iterator n-steps ahead. If n is None, consume entirely."
    # Use functions that consume iterators at C speed.
    if n is None:
        # feed the entire iterator into a zero-length deque
        collections.deque(iterator, maxlen=0)
    else:
        # advance to the empty slice starting at position n
        next(islice(iterator, n, n), None)

Answer 6

Using Ignacio's solution.

import collections
def all_but_the_last_n(iterable, n):
    it = iter(iterable)
    fifo = collections.deque()
    for _, i in zip(range(n), it):
        fifo.append(i)
    for i in it:
        fifo.append(i)
        yield fifo.popleft()

print(list(all_but_the_last_n(range(10), 3)))
print(list(all_but_the_last_n('abcdefghijkl', 3)))

It is unfortunate that collections does not have a circular buffer. This would be more efficient from a cache miss standpoint with one.