Get sub tokens from tokens in ANTLR4 in Java

Question

My grammar is:

//parser
expression : SYNTAX;

//lexer
FIELD : [A-Za-z]+;
SYNTAX : '${'FIELD'}'

I want to extract FIELD from a string passed as input. For example, when I pass ${Test String} as input, I get one token '${Test String}', I want to extract 'Test String' out of it ie, the FIELD token. I do not want to change my grammar as suggested here . Can I also avoid using regex to extract 'Test String'. Does antlr4 provide any support to extract sub tokens?

Answer 1

There's nothing like sub-tokens in ANTLR. In your SYNTAX rule the FIELD rule serves as a fragment - essentially just a macro to be expanded at that place. There's no difference between your grammar and

//parser
expression : SYNTAX;

//lexer
FIELD : [A-Za-z]+;
SYNTAX : '${'[A-Za-z]+'}'

Both produce exactly same parse rand lexer.

If you're not willing to make it a parser rule, you'll have to extract it yourself (regex or otherwise).