How can I use a JFlex lexer with JavaCC parser?

Question

I'm trying to learn how to use JavaCC to write a parser.

I have already generated a lexer using JFlex, and that returns a list of tokens. Each token is it's own class.

I'm writing the production rules, but for instance, I can't write ";" because it won't be receiving a semicolon, but an instance of TokenSemicolon instead?

What can I do?

In addition, I'm confused about the TokenMangager, etc. I already have a lexer and my own list of compatible token classes. What is this?

Please help because I am very confused.

Answer 1

You ask two related questions:

"What is this?" By which, I assume you mean: "What is the token manager?"

The token manager is a source of token objects. Every JavaCC parser needs a source of tokens. The tokens, by the way, are represented by objects of class Token . There are two ways to make a token manager.

1. Let JavaCC generate one for you. JavaCC generates a lexical analyzer based on a set of rules that you put in the .jj file. In this way it is much like JFlex. This is the default.
2. Write your own. To do this set option USER_TOKEN_MANAGER=true . Then JavaCC will generate a Java interface called TokenManager . All you need to do is to implement that interface with your own class. Of course you should then construct the parser using an object of that class.

"What can I do?"

There are a few possibilities.

1. Rewrite your JFlex code in JavaCC. Then the JavaCC generated token manager will do essentially the same thing as your JFlex lexer, but it will implement the right interface and it will generate tokens of the appropriate type (ie Token .)
2. Write an adapter class. Use JavaCC's USER_TOKEN_MANAGER=true option and write an adapter class that wraps your JFlex and implements the TokenManager interface.
3. Convince JFlex to generate a lexer that can be used with JavaCC. I'm not sure whether this is even possible, but if it is, it might be the best option. In this case you'd use USER_TOKEN_MANAGER=true . Then make a class: class FooLexer extends FooJLexLexer implements TokenManager { ...put constructors here... }

For option 3 you'd have to ensure that the generated lexer actually implements all the methods required by TokenManager . If you really need all your own token classes, you could have them extend the generated Token class.

If you go with option 2, your code to build the parser will likely look a little like this

TokenManager tm = new AdaptJFlexLexerToJavaCC( jflexLexer ) ;
FooParser p = new FooParser( tm ) ;

Option 3 is tempting to try. It might be the easiest if it works out.

If Option 3 doesn't work out and unless there are compelling reasons to keep the JFlex lexer, I'd go for option 1. The translation from JFlex to JavaCC is likely to be largely mechanical and therefore quick and easy. The only thing in JFlex that JavaCC doesn't have a good solution for is the A / B construct.

Whichever option you take, keep in mind that JavaCC expects every Token to have a .kind field. This is an integer, but you will find symbolic names for the integers in the generated interface FooConstants .