ANTLR4 中的字符串文字

Question

I'm using antlr4 C++ runtime and I'd like to create a string literal in my lexer definition file. How can I do this?

What I have so far:

V_STRING            :   '"' ~('\\' | '"')* '"';

I doesn't work with

printf("string literal\n");

but works with

printf("string literal\\n");

I don't want to explicitly escape the new line character.

my assumptions are that antlr interprets the new line character as a regular new line (when reading a file, for example).

Thanks in advance.

Answer 1

It's always a good idea to list out your token stream to see if your Lexer rules really do what you expect. (Look into the tokens option of the TestRig ; also, some plugins will show you your tokens)

In your case your rule essentially says that a String is " a " followed by 0 or more characters that are not a \ or a " and then a " ".

So, when the Lexer encounters your \ , matches the ~('\\\\'|'")* part of the rule and then looks for a " (which it does not find, since the \ is followed by a n ), so It won't recognize "string literal\n" as a V_STRING token (it also fails to match "string literal\\n" as well, here, so I'm not quite sure what's going on with the example that "works").

try:

V_STRING: '"' ~["]* '"';

Note: this is a very simple String rule, but it accepts your input. You probably want to examine grammars for other languages to see how you might want to handle strings in your language; there are several approaches (and many of them involve using Lexer modes). You can find examples here)

If you want the "\n" to be treated as a newline, just understand that the parser won't do that for you, you'll just see the characters "" and "n". It'll be up to you to handle encoding the escaped characters (and it's once you try to handle " that it'll get more complicated and you'll need to look into Lexer modes)