antlr4 Similar token definition

Question

I've a problem with tokens definition.

here is my grammar.

r: PROPNAME ':' PROPVALUE
PROPNAME: [a-zA-Z]+
PROPVALUE: [a-zA-Z0-9]+ 

if I use

name:christof123 it match

if I use

name:christof it doesn't match

Arguing 'christof' is PROPNAME lexer when PROPVALUE waited since 'christof' matches both PROPVALUE & PROPNAME expressions.

But I don' want matching on

name123:christof

Any Idea?

Answer 1

Like you said, the lexer will match christof with PROPNAME because that comes first in your definition that will match the longest. You can check the matches using grun .

antlr4 MyGrammer.g4
javac -g *.java
grun MyGrammer r -tokens
# enter your input string and press ctlr+d

Your grammar produces the below matches which gives the error.

name:christof
line 1:13 token recognition error at: '\n'
[@0,0:3='name',<2>,1:0]
[@1,4:4=':',<1>,1:4]
[@2,5:12='christof',<2>,1:5]
[@3,14:13='<EOF>',<-1>,2:0]
line 1:5 mismatched input 'christof' expecting PROPVALUE

So modifying your grammar to the below would solve.

r: name ':' value;

name: ALPHA;
value: ALPHA | ALPHANUM;

ALPHA: [a-zA-Z]+;
ALPHANUM: [a-zA-Z0-9]+;

Which produces the following match with grun.

name:christof
line 1:13 token recognition error at: '\n'
[@0,0:3='name',<2>,1:0]
[@1,4:4=':',<1>,1:4]
[@2,5:12='christof',<2>,1:5]
[@3,14:13='<EOF>',<-1>,2:0]