Parsing python with PLY, how to code the indent and dedent part

Question

I was trying to parse the function definition for the python language with PLY. I am encountering issues related to the indentation. For instance for a for statement, I would like to be able to know when the block ends. I read the python grammar here: http://docs.python.org/2/reference/grammar.html And the grammar for this part is:

for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT

I don't know how to describe the INDENT and DEDENT tokens with PLY. I was trying something like:

def t_indentation(t):
    r'    |\t'
    #some special treatment for the indentation.

But it seems that PLY consider that regexes with spaces match the empty string and does not build the lexer... Even if I would have managed to have the INDENT token I am not sure about the way to get the DEDENT one...

Is there a way to do that with PLY?

Answer 1

You have to use states to parse INDENT and UNDENT.

example of parsing python like language

Answer 2

PLY includes in its examples one for a subset of Python to demonstrate how to handle indentation:

https://github.com/dabeaz/ply/blob/master/example/GardenSnake/GardenSnake.py