I am currently writing a compiler and I\'m in the Lexer phase.
I know that the lexer tokenizes the input stream.
However, consider the following stream:
There is no real gain to have "letter" as an intermediate step - instead "foo" should probably be an identifier. Otherwise you could understand int as "letter letter letter" too, which doesn't make much sense.
int