ANTLR: Lexer does not recognize token

问题

Given the following Lexer grammar:

lexer grammar CodeTableLexer;

CodeTabHeader   : '[code table 1.0]';
Code            : 'code';
Table           : 'table';
End             : 'end';
Row             : 'row';
Naming          : 'naming';
Dfltlang        : 'dfltlang';
Language        : 'english' | 'german' | 'french' | 'italian' | 'spanish';
Null            : 'null';

Number
    : Int ('.' Digit*)?
    ;

Identifier
    : ('a'..'z' | 'A'..'Z' | '_') ('a'..'z' | 'A'..'Z' | '_' | '$' | '.' | Digit)*
    ;

String
@after {
    setText(getText().substring(1, getText().length() - 1).replaceAll("\\\\(.)", "$1"));
}
    : '"' (~('"'))* '"'
    ;

Comment
    : '--' ~('\r' | '\n')* { skip(); }
    | '/*' .* '*/'         { skip(); }
    ;

Space
    : (' ' | '\t' | '\r' | '\n' | '\u000C') { skip(); }
    ;

fragment Int
    : '1'..'9'
    | '0'
    ;

fragment Digit
    : '0'..'9'
    ;

... the following parser grammar:

parser grammar CodeTableParser;

parse
    : header^ begin (row)* end EOF
    ;

header
    : CodeTabHeader
    ;

begin
    : Code Table Identifier row
    ;

row
    : Row (Number | Identifier) value
    ;

value
    : (Identifier (Number | Identifier | String))+
        (Naming Dfltlang String (Language String)*)?
    ;

end
    : End Code Table
    ;

... and finally the following source to parse:

[code table 1.0]

code table my_table
row 1
    id              "my_id_1"
    name            "my_name_1"
    descn           "my_description_1"
    naming
        dfltlang    "My description 1"
        english     "My description 1"
        german      "Meine Beschreibung 1"

row 2
    id              "my_id_2"
    name            "my_name_2"
    descn           "my_description_2"
    naming
        dfltlang    "My description 2"
        english     "My description 2"
        german      "Meine Beschreibung 2"

end code table

I always get error message

line 1:0 extraneous input '[code table 1.0]' expecting CodeTabHeader

I’m a bit confused… token CodeTabHeader is defined as ‘[code table 1.0]’ in my lexer; am I missing something? Any help would be really appreciated.

Thanks, J3d

回答1:

If you split the lexer and parser in two different files, explicitly indicate in the options-block of your parser which tokens it should use:

options {
  tokenVocab=CodeTableLexer;
}

来源：https://stackoverflow.com/questions/12603326/antlr-lexer-does-not-recognize-token

标签

antlr

antlr3

lexer

parser-generator