自定义键值对的正则表达式优化

Question

I am trying to extract some key-value pairs plus their preceding text from a large file, but the regular expression used runs very slowly, so it needs optimization.

The input consists of fairly short strings with 1 or 2 key-value pairs, like

one two three/1234==five/5678 some other text

or

one two three/1234==five/5678 some other text four/910==five/1112 more text

The (apparently suboptimal) regular expression used is

(.*?)\\s*([^ /]+)\\s*/\\s*([\\d]+)\\s*==\\s*([^ /]+)\\s*/\\s*([\\d]+)\\s*

(Spaces may appear in numerous areas within the string, hence the repeated \\s* elements.)

Sample code to test the above:

  public static void main(String[] args) {
    String text = "one two three/1234==five/5678 some other text";
    text = "one two three/1234==five/5678 some other text four/910==five/1112 more text";
    String regex = "(.*?)\\s*([^ /]+)\\s*/\\s*([\\d]+)\\s*==\\s*([^ /]+)\\s*/\\s*([\\d]+)\\s*";
    Matcher matcher = Pattern.compile(regex).matcher(text);
    int end = 0;
    System.out.println("--------------------------------------------------");
    while (matcher.find()) {
      System.out.println("\"" + matcher.group(1) + "\"");
      System.out.println(matcher.group(2) + " == " + matcher.group(3));
      System.out.println(matcher.group(4) + " == " + matcher.group(5));
      end = matcher.end();
      System.out.println("--------------------------------------------------");
    }
    System.out.println(text.substring(end).trim());
  }

The output is the key-value pairs, plus the preceding text (all extracted fields are required). For example, for the longer string, the output is:

--------------------------------------------------
"one two"
three == 1234
five == 5678
--------------------------------------------------
"some other text"
four == 910
five == 1112
--------------------------------------------------
more text

In other words, the matcher.find() method runs for 1 or 2 rounds, depending on whether the string has the short or long form (1 or 2 key-value pairs, respectively).

The problem is that the extraction speed is low and at times, depending on the variation of the input string, the find() method takes a lot of time to complete.

Is there any better form for the regular expression, to significantly speed up processing?

Answer 1

It's never a good idea to put (.*?) at the beginning of a regex.

First, it can be slow. Although in theory non-greedy matches can be handled efficiently (see, for example, Russ Cox's re2 implementation), many regex implementations do not handle non-greedy matches very well, especially in the case where the find operation is going to fail. I don't know whether the Java regex implementation falls into this category or not, but there's no reason to tempt fate.

Second, it's pointless. The semantics of regex searching is that the first possible match will be found, which is identical to the semantics of .*? . To get the capture (.*?) , you only need the substring from the end of the previous match (or the beginning of the string) to the beginning of the current match. That's trivial, especially since you're already tracking the end of the previous match.

Answer 2

How are you reading the file? If you read the file line-by-line with BufferedReader#readLine() or Scanner#nextLine() , all you need to do is add \\G to the beginning of your regex. It acts like \\A the first time you apply the regex, anchoring the match to the beginning of the string. If that match succeeds, the next find() will be anchored to the position where the previous match ended. If it doesn't find a match starting right there , it gives up and doesn't look for any more matches in that string.

EDIT: I'm assuming each of the sequences you want to match, whether it's one key/value pair or two, is on its own line. If you read the file one line at a time, you can run the code in your question on on each line.

As for why your regex is so slow, it's because the regex engine has to make multiple match attempts--possibly hundreds of them--on every non-matching line before it gives up. It isn't smart enough to realize that if the first attempt on a given line fails, no further attempts on that line will do any good. So it bumps forward one position and tries again. And it keeps doing that for the whole line.

If you were only expecting one match per line, I would say to use a start-of-line anchor ( ^ in MULTILINE mode).