JavaCC：如何维护原始文本（带空格）

Question

Let's assume that I have a simple JavaCC grammar to parse additions and subtractions:

....
void CompilationUnit() :
{}
{
  (Expression())+
  EOF
}
void Expression() :
{}
{
  Number()
  (
    Addition()
  | Subtraction()
  )*
}
void Number() :
{}
{
  
}
void Addition() :
{}
{
   Number()
}
void Subtraction() :
{}
{
   Number()
}

I have classes that are using the AST produced by this grammar to calculate the result:

public class Calculator extends DepthFirstVisitor {
  int result = -1;
  public void visit(Expression n) {
    if (result >= 0) System.out.println(toText(n) + " = " + result);
    result = 0;
    super.visit(n);
  }
  public void visit(Number n) {
    ...
  }
  public void visit(Addition n) {
    ...
  }
  ....
}

I am able to calculate the value of the expression but I also need the original expression as well (as it appeared). So for the following input:

5 + 2 - 1
  2 + 1

I want to have the following output:

5 + 2 - 1 = 6
2 + 1 = 3

Unfortunately, because I'm skipping characters like spaces or newlines, what I'm getting is:

5+2-1 = 6
2+1 = 3

Is there any way I can output the original text (including the skipped characters)?

Please note that the actual problem is much bigger and the grammar much more complicated. So I'm not really looking for a solution specific to the above problem (eg preprocess the lines and split them on newline characters or modify methods to "manually" add spaces after every token) but more like a solution that is using some JavaCC feature.

Answer 1

Both ANTLr and Xtext support "hidden tokens" for whitespace and comments. See here for some hints or use Google with that term. Perhaps JavaCC has some similar concept.

EDIT : JavaCC seems to use the term "special token". See here for some details .

Answer 2

Basically you can't do this in a compiler. You would have to capture whitespace as a token in the grammar and allow it everywhere it is allowed, which is everywhere, and the resultant grammar would be so complex as to be infeasible to implement or maybe even generate. You will have to make do with capturing a reference to the co-ordinates in the source code (line and column) where the entity came from: maybe for example the text of the current line and column number.

There's a reason why compilers behave the way they do.