Using lookahead with generators
I have implemented a generator-based scanner in Python that tokenizes a string into tuples of the form (token type, token value) : for token in scan("a(b)"): print token would print ("literal", "a") ("l_paren", "(") ... The next task implies parsing the token stream and for that, I need be able to look one item ahead from the current one without moving the pointer ahead as well. The fact that iterators and generators do not provide the complete sequence of items at once but each item as needed makes lookaheads a bit trickier compared to lists, since the next item is not known unless __next__()