用于自定义需求的Antlr解析器

Question

I have a very peculiar requirement to parse inputs using ANTLR. I would like to be able to parse expressions like;

Correct Inputs

• user name
• user_name user-name
• | EATALL any thing could come here/ok | EATALL ...

Invalid Inputs

• user/name
• user&name^face

Well, any expressions which come after | EATALL | EATALL & before | EATALL | EATALL (if any) must be obtained as a single token. While in case of other simple inputs where | EATALL | EATALL doesn't appear, only valid combination of _ , - , [a-zA-Z0-9] is tokenized as a one token. In pseudocode,

• user name -> [user] [name]
• user_name -> [user_name]
• |EATALL user/name my user -> [user/name my user]

This already seems like an ambiguous case of tokenization for me. I am seeking your suggestions on dealing problems like these in antlr. Thanking you in advanced.

Answer 1

So, what have you tried? Is you question specific to Antlr 3 or can you use Antlr 4?

For Antlr 3, you can use semantic predicates to condition token rule selection. Since Antlr 4 does not have symbolic semantic predicates, you can use native code actions to achieve essentially the same result. For example (untested):

lexer grammar eatall ;

ValidSimple : { isCurrenLineJustTEXTandWS() }? TEXT ;
-- or --
ValidSimple : TEXT ( WS TEXT)* EOL?  { emitEachTEXTasNewValidSimpleToken(); } ;

ValidEatAll : IgnoreL .*? IgnoreR    { trimIgnoreLRTextfromTokenText(); } ;
Invalid     : WS+ | .*? EOL?         -> channel(HIDDEN) ;

IgnoreL : .*? MARK ;
IgnoreR : MARK .*? EOL? ;

fragment MARK : '| EATALL' ;
fragment TEXT : [a-zA-Z0-9_-] ;
fragment EOL  : '\r'? '\n' ;
fragment WS   : [ \t] ;