Pattern.matches（）给出了StackOverflowError

Question

I'm using java's Pattern.matches to match a block of data to a regex. The block of data can be a single line or multiple lines. The problem is that once my data becomes more than 15 lines (typically more than 17-18 lines), i start getting stackoverflowerror. For data less than 15 lines the regex works fine.

The Regex is of this format:
domainname -> space -> , -> space -> number -> space -> , -> space -> number -> newline

String regex = "^(([a-zA-Z0-9][a-zA-Z0-9\\-]*\\.)+([a-zA-Z]{2,})\\s*,\\s*\\d+\\s*,\\s*\\d+(\\r?\\n)?)+$";

The data block i use to test against this regex is this

abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456
abc.com, 123, 456

This is the code:

String regex = "^(([a-zA-Z0-9][a-zA-Z0-9\\-]*\\.)+([a-zA-Z]{2,})\\s*,\\s*\\d+\\s*,\\s*\\d+(\\r?\\n)?)+$";
boolean valid = Pattern.matches(regex, data); //fails here

Answer 1

I can't tell you the reason for this error; the regex itself is fine and not subject to catastrophic backtracking or any other obvious error.

Perhaps you can reduce the number of backtracking positions the regex engine saves by using possessive quantifiers ( ++ instead of + , *+ instead of * , {2,}+ instead of {2,} etc.). Also, you don't need the capturing groups (thanks Thomas), so I've changed them into non-capturing ones:

"(?:(?:[a-zA-Z0-9][a-zA-Z0-9-]*+\\.)++([a-zA-Z]{2,}+)\\s*+,\\s*+\\d++\\s*+,\\s*+\\d++(\r?+\n)?+)++"

This won't change the behaviour of the regex (except for the removal of the unnecessary anchors since you're using Pattern.matches() ), but perhaps it helps avoid StackOverflows. I don't have a Java SDK installed, so I can't test it myself, though.

Answer 2

You might try and use atomic groups ( (?>expression) ) to prevent backtracking:

Here's a test that failed with a block of 1000 lines using your regex but succeeds now (takes a while, thus I only tested with 5000 20000 :) ):

String regex = "(?>(?>[a-zA-Z0-9][a-zA-Z0-9\\-]*\\.)+(?>[a-zA-Z]{2,})\\s*,\\s*\\d+\\s*,\\s*\\d+(?>\\r?\\n)?)+";

StringBuilder input = new StringBuilder();

for( int i = 0; i < 1000000; ++i) {
  input.append("abc.com, 123, 456\n");
}

Pattern p = Pattern.compile( regex );
Matcher m = p.matcher( input );

System.out.println(m.matches());

So after all, it might still be a backtracking problem.

Update : just let that test run with 20000 lines and still didn't fail. That's at least 20 times as much as before. :)

Update 2 : looking at my test again I found the slow part, the string concatenation. (o..O). I've updated the test and used 1 Million lines, still no fail. :)

Answer 3

The problem is that your regex is too complicated. Each line of input that you process results in (I think) 10 backtrack points, and at least some of these seem to be handled by the regex engine recursing. That could be a few hundred stack frames which would be enough to give you StackOverflowError .

IMO, you need to modify the pattern so that it will match one group / line of data. Then call Matcher.find repeatedly to parse each line. I expect that you will find that this is faster.

---

Optimizing the regex in other ways while still trying to match the entire block in one go probably won't work. You may be able to get it to match N times more lines of data, but as you increase the number of lines in the input you are likely to run into the same problem again.

And even if you do get it to work as a multi-line regex, there is a chance that it won't work with other implementations of the Java regex libraries; eg in older Oracle JREs or non-Oracle implementations.

---

I agree with the other answers that this is not an example of "catastrophic backtracking". Rather it is an interaction between the way that the regex engine handles backtrack points, and the fact that there are simply too many of them when you give it multiple lines of input.

Answer 4

I've reproduced this problem, but only for much larger strings.

$ java -version
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10.2)    (6b22-1.10.2-0ubuntu1~11.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)

My test code:

public class Testje
{
    public static void main(String... args)
    {
        String regex = "^(([a-zA-Z0-9][a-zA-Z0-9\\-]*\\.)+([a-zA-Z]{2,})\\s*,\\s*\\d+\\s*,\\s*\\d+(\\r?\\n)?)+$";
        String data = "";
        for (int i = 0; i<224; i++) data += "abc.com, 123, 456\n";
        System.out.println(data.matches(regex));
    }
}

For anything smaller than 224 in that for loop, the code runs fine. For 224 and more copies of that line, I get a huge stack trace.

Oh, note that using (?: groups does not change the size of the string that still works.