Java解析器和正则表达式

Question

I'm writing a parser, and I'm currently working on matching different tokens, and I'm having a bit of trouble with matching. I have a test file:

while a != b
  if a > b
    a := a - b
  if a <= b
    b := b - a
elihw

And part of my code:

private static Scanner sc = new Scanner(System.in);
private static Pattern tokenPattern = Pattern.compile("[ ]+");
private static Pattern idPattern = Pattern.compile("[a-zA-Z]+");

....main(...) {
      sc.useDelimiter(tokenPattern);
      statement();
    }

public static void statement() {
    System.out.println("Statement");
    String token = null;
    while (sc.hasNext()) {
        if (sc.hasNext(idPattern)) {
            token = sc.next();
            System.out.print(" (" + token + ") ");
        }
        else {
            token = sc.next();
            System.out.print(token + ' ');
        }
    }
}

When I run this method, it matches the strings before the operators, but not the ones after. The parenths are there just to mark the ones it matches. For example, the line

a := a - b

will produce output:

(a) := (a) - b

I cannot figure out why the b is not matched.

Also if anyone could help me with a regex that matches operators that would be great. I have tried many variations of things like this:

[\+\-\*\\]
[\\+\\-\\*\\\]
[+][-][*][/]

But cannot seem to get it right.

Answer 1

The scanner is likely including a non-printable (newline) character as part of the token.

Try this:

private static Pattern tokenPattern = Pattern.compile("[ \r\n\t]+");

For the operators, try this:

[<>+-/*=:]+