如何从在ANTLR中已标记化的代码中获取ID Lexer

Question

I have a lexer class that determined the token ID Lexer. The code is :

public class Antlr3JavaLexer extends Lexer {
public static final int PACKAGE=84;
public static final int PUBLIC=87;
public static final int STATIC=90;
public static final int IDENT=164;
public static final int CLASS = 70;  
}

Now I have a java class named hello.java that will be generated by ANTLR. The code is like this public class hello{ public static void main(String args[]){ System.out.print("Hello World");} }

now, time to ANTLR lexer that i created get shot..

BufferedReader in = null;
try {
        in = new BufferedReader(new FileReader(mainFile.getAbsolutePath())); // Assumption this is to read the hello.class
    } catch (FileNotFoundException e1) {
        // TODO Auto-generated catch block
        e1.printStackTrace();
    }

    final Antlr3JavaLexer lexer = new Antlr3JavaLexer();

    try {
        lexer.setCharStream(new ANTLRReaderStream(in));
    } catch (IOException e) {
        e.printStackTrace();
        // return false;
    }

    final CommonTokenStream tokens = new CommonTokenStream();
    tokens.setTokenSource(lexer);

    Antlr3JavaParser parser = new Antlr3JavaParser(tokens); 
    System.out.println(tokens + "and" + "\n"); //First Print
    System.out.println(tokens.getTokens); // Second Print

Lucky, I get the output like this : https://www.dropbox.com/s/tsogz10eouo9f9h/ID%20Token.bmp

So, the question is : How can I get the ID from the tokens?. For example public class hello is public = 87, class = 70, hello is identifier = 164. SO, perhaps the output like this

8770164 

Thanks 4 the help...

Answer 1

When you append an object to a StringBuilder (or StringBuffer , which you should probably not be using), it calls ToString() on the object and appends the resulting text. If you want to format the list in another way, you'll need to iterate over the elements and append the text in the desired format.

In this case, the desired format appears to be Token.getType() .

boolean first = true;
for (Token token : tokens.getTokens()) {
  if (first) {
    first = false;
  } else {
    sb.append(", ");
  }

  sb.append(token.getType());
}