Dynamically create lexer rule

问题

Here is a simple rule:

NAME : 'name1' | 'name2' | 'name3';

Is it possible to provide alternatives for such rule dynamically using an array that contains strings?

回答1:

Yes, dynamic tokens match IDENTIFIER rule

In that case, simply do a check after the Id has matched completely to see if the text the Id matched is in a predefined collection. If it is in the collection (a Set in my example) change the type of the token.

A small demo:

grammar T;

@lexer::members {
  private java.util.Set<String> special;

  public TLexer(ANTLRStringStream input, java.util.Set<String> special) {
    super(input);
    this.special = special;
  }

}

parse
 : (t=. {System.out.printf("\%-10s'\%s'\n", tokenNames[$t.type], $t.text);})* EOF
 ;

Id
 : ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '0'..'9')*
   {if(special.contains($text)) $type=Special;}
 ;

Int
 : '0'..'9'+
 ;

Space
 : (' ' | '\t' | '\r' | '\n') {skip();}
 ;

fragment Special : ;

And if you now run the following demo:

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    String source = "foo bar baz Mu";
    java.util.Set<String> set = new java.util.HashSet<String>();
    set.add("Mu");
    set.add("bar");
    TLexer lexer = new TLexer(new ANTLRStringStream(source), set);
    TParser parser = new TParser(new CommonTokenStream(lexer));
    parser.parse();
  }
}

You will see the following being printed:

Id        'foo'
Special   'bar'
Id        'baz'
Special   'Mu'

ANTLR4

For ANTLR4, you can do something like this:

grammar T;

@lexer::members {
  private java.util.Set<String> special = new java.util.HashSet<>();

  public TLexer(CharStream input, java.util.Set<String> special) {
    this(input);
    this.special = special;
  }
}

tokens {
  Special
}

parse
 : .*? EOF
 ;

Id
 : [a-zA-Z_] [a-zA-Z_0-9]* {if(special.contains(getText())) setType(TParser.Special);}
 ;

Int
 : [0-9]+
 ;

Space
 : [ \t\r\n] -> skip
 ;

test it with the class:

import org.antlr.v4.runtime.*;
import java.util.HashSet;
import java.util.Set;

public class Main {

  public static void main(String[] args) {

    String source = "foo bar baz Mu";
    Set<String> set = new HashSet<String>(){{
      add("Mu");
      add("bar");
    }};

    TLexer lexer = new TLexer(CharStreams.fromString(source), set);
    CommonTokenStream tokenStream = new CommonTokenStream(lexer);
    tokenStream.fill();

    for (Token t : tokenStream.getTokens()) {
      System.out.printf("%-10s '%s'\n", TParser.VOCABULARY.getSymbolicName(t.getType()), t.getText());
    }
  }
}

which will print:

Id         'foo'
Special    'bar'
Id         'baz'
Special    'Mu'
EOF        '<EOF>'

来源：https://stackoverflow.com/questions/9008134/dynamically-create-lexer-rule