How do I Handle tokens within text blocks in JavaCC?

Question

I have a simple aspect of a DSL where I can define a key and a value as such:

 mykey=\\ This is my $REF{useful} multiline string where I terminate with a backslash but I support escaped \\\\ characters and I wish to handle the value part of this string as 3 blocks in this example. \\ 

The three tokens (for the value part) I would like in this example are

• ValueLiteral == This is my
• ValueReference == $REF{useful}
• ValueLiteral == multiline etc....

I defined a rule for the value as such:

void multiLineValue(): {} {
  < BACKSLASH >< EOL >
  (
    valuePartLiteralMulti() |
    valuePartRef()
  )*
  < BACKSLASH >
}

Here is my TOKEN definition for the multiline string type:

TOKEN :
{
     < MULTILINE_STRING:(  ( (~["\\"])
    | ("\\"
        ( ["\\", "'", "\"", "$", "n", "r", "t", "b", "f"]
        | ["u", "U"]["+"]["0"-"9","a"-"f","A"-"F"]["0"-"9","a"-"f","A"-"F"]["0"-"9","a"-"f","A"-"F"]["0"-"9","a"-"f","A"-"F"]
        )
      ) ))+>
}

My problem is that my multi line string token type also consumes the character sequence of the '$REF{' characters.

I would like to modify this multi-line string so that it will stop consuming characters when it encounters an unescaped "$REF{" (but will continue reading past a "\\$REF{" sequence).

Any assistance would be much appreciated.

Answer 1

I'm not sure, but in your token definition you also include $ (in unicode?), maybe you should add ~("$") (or the unicode equivalent) at the beginnig.

Or you can use syntatic LOOKAHEAD, something like LOOKAHEAD(valuePartRef())...

ps Can you have more than one REF?