Using Antlr for parsing data from never-ending stream

Question

Is Antlr suitable for parsing data from streams that don\'t have EOF right after the text to parse? According to my observation, the lexer does not emit the current token until

Answer 1

The ANTLR 4 book was originally going to contain an example of parsing a streaming input, but I argued against it due to the severe complications that will inevitably arise from the use of an adaptive unlimited lookahead parser for something like this.

ANTLR 4 has no guaranteed lookahead bound (and no way to tell it to look for or even attempt to enforce one), so any implementation that operates on a blocking stream has the possibility of deadlock without returning information about the parse leading up to that point. I wouldn't even entertain the possibility of parsing a streaming input unless I saw an intermediate buffer in place first.

1. Take all available (or previously unparsed) input and place it in a String or char[].
2. Create an ANTLRInputStream for the buffer.
3. Attempt to lex/parse this stream, which will have an implicit EOF on the end.

The result of the parse will tell you whether to discard the results to that point, or hold on to them to retry when more data is available:

• If no syntax error occurs, the input was successfully parsed, and you can parse the next section of input when it becomes available later.

• If a syntax error is reported before the EOF token is consumed, then a syntax error appears in the actual input so you'll want to handle it (report it to the user, etc...).

• If a syntax error is reported at the point where the EOF token is consumed then additional input may resolve the problem - ignore the results of the current parse, and then retry once more data is available from the input stream.