How can I modify the text of tokens in a CommonTokenStream with ANTLR?

Question

I\'m trying to learn ANTLR and at the same time use it for a current project.

I\'ve gotten to the point where I can run the lexer on a chunk of code and output it to a C

Answer 1

The other given example of changing the text in the lexer works well if you want to globally replace the text in all situations, however you often only want to replace a token's text during certain situations.

Using the TokenRewriteStream allows you the flexibility of changing the text only during certain contexts.

This can be done using a subclass of the token stream class you were using. Instead of using the CommonTokenStream class you can use the TokenRewriteStream.

So you'd have the TokenRewriteStream consume the lexer and then you'd run your parser.

In your grammar typically you'd do the replacement like this:

/** Convert "int foo() {...}" into "float foo();" */
function
:
{
    RefTokenWithIndex t(LT(1));  // copy the location of the token you want to replace
    engine.replace(t, "float");
}
type id:ID LPAREN (formalParameter (COMMA formalParameter)*)? RPAREN
    block[true]
;

Here we've replaced the token int that we matched with the text float. The location information is preserved but the text it "matches" has been changed.

To check your token stream after you would use the same code as before.