JavaCC lexer doesn't work as expected (whitespace not ignored)

Question

I'm trying to implement a parser for the example file listed below. I'd like to recognize quoted strings with '+' between them as a single token. So I created a jj file, but it doesn't match such strings. I was under the impression that JavaCC is supposed to match the longest possible match for each token spec. But that doesn't seem to be case for me.

What am I doing wrong here? Why isn't my <STRING> token matching the '+' even though it's specified in there? Why is whitespace not being ignored?

options {
  TOKEN_FACTORY = "Token";
}

PARSER_BEGIN(Parser)

package com.example.parser;

public class Parser {

  public static void main(String args[]) throws ParseException {

      ParserTokenManager manager = new ParserTokenManager(new SimpleCharStream(Parser.class.getResourceAsStream("example")));
      Token token = manager.getNextToken();
      while (token != null && token.kind != ParserConstants.EOF) {
          System.out.println(token.toString() + "[" + token.kind + "]");
          token = manager.getNextToken();
      }

      Parser parser = new Parser(Parser.class.getResourceAsStream("example"));
      parser.start();
  }

}

PARSER_END(Parser)

// WHITE SPACE
<DEFAULT, IN_STRING_KEYWORD>
SKIP :
{
  " " // <-- skipping spaces
| "\t"
| "\n"
| "\r"
| "\f"
}

// TOKENS
TOKEN :
{
< KEYWORD1 : "keyword1" > : IN_STRING_KEYWORD
}

<IN_STRING_KEYWORD>
TOKEN : {<STRING : <CONCAT_STRING> | <UNQUOTED_STRING> > : DEFAULT 
| <#CONCAT_STRING : <QUOTED_STRING> ("+" <QUOTED_STRING>)+ >
// <-- CONCAT_STRING never matches   "+" part when input is "'smth' +", because whitespace is not ignored!?
| <#QUOTED_STRING : <SINGLEQUOTED_STRING> | <DOUBLEQUOTED_STRING> >
| <#SINGLEQUOTED_STRING : "'" (~["'"])* "'" >
| <#DOUBLEQUOTED_STRING : 
    "\""
      (
        (~["\"", "\\"]) |
        ("\\" ["n", "t", "\"", "\\"])
      )* 
    "\""
  >
| <#UNQUOTED_STRING : (~[" ","\t", ";", "{", "}", "/", "*", "'", "\"", "\n", "\r"] | "/" ~["/", "*"] | "*" ~["/"])+ >
}

void start() :
{}
{
  (<KEYWORD1><STRING>";")+ <EOF>
}

Here's an example file that should get parsed:

keyword1 "foo" + ' bar';

I'd like to match the argument of the first keyword1 as a single <STRING> token.

Current output:

keyword1[6]
Exception in thread "main" com.example.parser.TokenMgrError: Lexical error at line 1, column 15.  Encountered: " " (32), after : "\"foo\""
    at com.example.parser.ParserTokenManager.getNextToken(ParserTokenManager.java:616)
    at com.example.parser.Parser.main(Parser.java:12)

I'm using JavaCC 5.0.

Answer 1

STRING is expanding to the longest sequence that can be matched, which is "foo" as the error indicates. The space after the closing double quote is not part of the definition of the private token CONCAT_STRING . Skip tokens do not apply within the definition of other tokens, so you must incorporate the space directly into the definition, on either side of the + .

As an aside, I recommend have a final token definition like so:

<each-state-in-which-the-empty-string-cannot-be-recognized>
TOKEN : {
    < ILLEGAL : ~[] >
}

This prevents TokenMgrError s from being thrown and makes debugging a bit easier.