While using the python library rply, I get an unexpected token error when parsing more than one line. How can I fix this?

问题

For the practice, I decided to work on a simple language. When only a single line, my say(); command works fine, but when I do two says in a row, I get an error.

For Parsing I'm using rply. I was following this (https://blog.usejournal.com/writing-your-own-programming-language-and-compiler-with-python-a468970ae6df) guide. I've searched extesively but I cant find a solution.

This is the python code:

from rply import ParserGenerator
from ast import Int, Sum, Sub, Say, String


class Parser():
   def __init__(self):
        self.pg = ParserGenerator(
            # A list of all token names accepted by the parser.
            ['INTEGER', 'SAY', 'OPEN_PAREN', 'CLOSE_PAREN',
             'SEMI_COLON', 'SUM', 'SUB', 'STRING']
        )

    def parse(self):
        @self.pg.production('say : SAY OPEN_PAREN expression CLOSE_PAREN SEMI_COLON')
        def say(p):
            return Say(p[2])

        @self.pg.production('expression : expression SUM expression')
        @self.pg.production('expression : expression SUB expression')
        def expression(p):
            left = p[0]
            right = p[2]
            operator = p[1]
            if operator.gettokentype() == 'SUM':
                return Sum(left, right)
            elif operator.gettokentype() == 'SUB':
                return Sub(left, right)

        @self.pg.production('expression : INTEGER')
        def int(p):
            return Int(p[0].value)

        @self.pg.production('expression : STRING')
        def string(p):
            return String(p[0].value)

        @self.pg.error
        def error_handler(token):
            raise ValueError("Ran into a %s where it wasn't expected" % token.gettokentype())

    def get_parser(self):
        return self.pg.build()

When I run my program with the input of:

say("yo");

It works fine and return yo. However, when I input:

say("yo");
say("yoyo");

I expect it to return yo yoyo, but instead I get this error:

C:\Users\gdog1\Desktop\proj\intparser.py:42: ParserGeneratorWarning: 4 
shift/reduce conflicts
  return self.pg.build()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.3.3\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:/Users/gdog1/Desktop/proj/main.py", line 20, in <module>
    parser.parse(tokens).eval()
  File "C:\Python27\lib\site-packages\rply\parser.py", line 60, in parse
    self.error_handler(lookahead)
  File "C:\Users\gdog1\Desktop\proj\intparser.py", line 39, in error_handler
    raise ValueError("Ran into a %s where it wasn't expected" % 
token.gettokentype())
ValueError: Ran into a SAY where it wasn't expected

回答1:

Your grammar describes a single command:

say : SAY OPEN_PAREN expression CLOSE_PAREN SEMI_COLON

So that is what the parser accepts.

If you want the input to consist of multiple commands, you need to write a grammar which describes that input:

program : 
program : program say

回答2:

As the error, says its with the below line:

raise ValueError("Ran into a %s where it wasn't expected" % token.gettokentype())

Change it as below and check:

raise ValueError('Ran into a %s where it wasn't expected' % (token.gettokentype()))

来源：https://stackoverflow.com/questions/54265650/while-using-the-python-library-rply-i-get-an-unexpected-token-error-when-parsin

标签

python

parsing

token