如何提高.NET正则表达式的性能？

Question

I have a regular expression which parses a (very small) subset of the Razor template language. Recently, I added a few more rules to the regex which dramatically slowed its execution. I'm wondering: are there certain regex constructs that are known to be slow? Is there a restructuring of the pattern I'm using that would maintain readability and yet improve performance? Note: I've confirmed that this performance hit occurs post-compilation.

Here's the pattern:

new Regex(
              @"  (?<escape> \@\@ )"
            + @"| (?<comment> \@\* ( ([^\*]\@) | (\*[^\@]) | . )* \*\@ )"
            + @"| (?<using> \@using \s+ (?<namespace> [\w\.]+ ) (\s*;)? )"

            // captures expressions of the form "foreach (var [var] in [expression]) { <text>" 
/* ---> */      + @"| (?<foreach> \@foreach \s* \( \s* var \s+ (?<var> \w+ ) \s+ in \s+ (?<expressionValue> [\w\.]+ ) \s* \) \s* \{ \s* <text> )"

            // captures expressions of the form "if ([expression]) { <text>" 
/* ---> */      + @"| (?<if> \@if \s* \( \s* (?<expressionValue> [\w\.]+ ) \s* \) \s* \{ \s* <text> )"  

            // captures the close of a razor text block
            + @"| (?<endBlock> </text> \s* \} )"

            // an expression of the form @([(int)] a.b.c)
            + @"| (?<parenAtExpression> \@\( \s* (?<castToInt> \(int\)\s* )? (?<expressionValue> [\w\.]+ ) \s* \) )"
            + @"| (?<atExpression> \@ (?<expressionValue> [\w\.]+ ) )"
/* ---> */      + @"| (?<literal> ([^\@<]+|[^\@]) )",
            RegexOptions.IgnorePatternWhitespace | RegexOptions.Multiline | RegexOptions.ExplicitCapture | RegexOptions.Compiled);

/* ---> */ indicates the new "rules" that caused the slowdown.

Answer 1

As you are not anchoring the expression the engine will have to check each alternative sub-pattern at every position of the string before it can be sure that it can't find a match. This will always be time-consuming, but how can it be made less so?

Some thoughts:

I don't like the sub-pattern on the second line that tries to match comments and I don't think it will work correctly.

I can see what you're trying to do with the ( ([^\\*]\\@) | (\\*[^\\@]) | . )* - allow @ and * within the comments as long as they are not preceded by * or followed by @ respectively. But because of the group's * quantifier and the third option . , the sub-pattern will happily match *@ , therefore rendering the other options redundant.

And assuming that the subset of Razor you are trying to match does not allow multiline comments, I suggest for the second line

+ @"| (?<comment> @\*.*?\*@ )"

ie lazily match any characters (but newlines) until the first *@ is encountered. You are using RegexOptions.ExplicitCapture meaning only named groups are being captured, so the lack of () should not be a problem.

I also do not like the ([^\\@<]+|[^\\@]) sub-pattern in the last line, which equates to ([^\\@<]+|<) . The [^\\@<]+ will greedily match to the end of the string unless it comes across a @ or < .

I do not see any adjacent sub-patterns that will match the same text, which are the usual culprits for excessive backtracking, but all the \\s* seem suspect because of their greed and flexibility, including matching nothing and newlines. Perhaps you could change some of the \\s* to [ \\t]* where you know you don't want to match newlines, for example, perhaps before the opening bracket following an if .

I notice that nhahtdh has suggested you use use atomic grouping to prevent the engine backtracking into the previously matched, and that is certainly something worth experimenting with as it is almost certainly the excessive backtracking caused when the engine can no longer find a match that is causing the slow-down.

What are you trying to achieve with the RegexOptions.Multiline option? You do not look to be using ^ or $ so it will have no effect.

The escaping of the @ is unnecessary.

Answer 2

As others have mentioned, you can improve the readability by removing unnecessary escapes (such as escaping @ or escaping characters aside from \\ inside a character class; for example, using [^*] instead of [^\\*] ).

Here are some ideas for improving performance:

Order your different alternatives so that the most likely ones come first.

The regex engine will attempt to match each alternative in the order that they appear in the regex. If you put the ones that are more likely up front, then the engine will not have to waste time attempting to match against unlikely alternatives for the majority of cases.

Remove unnecessary backtracking

Not the ending of your "using" alternative: @"| (?<using> \\@using \\s+ (?<namespace> [\\w\\.]+ ) (\\s*;)? )"

If for some reason you have a large amount of whitespace, but no closing ; at the end of a using line, the regex engine must backtrack through each whitespace character until it finally decides that it can't match (\\s*;) . In your case, (\\s*;)? can be replaced with \\s*;? to prevent backtracking in these scenarios.

In addition, you could use atomic groups (?> ... ) to prevent backtracking through quantifiers (eg * and + ). This really helps improve performance when you don't find a match. For example, your "foreach" alternative contains \\s* \\( \\s* . If you find the text "foreach var..." , the "foreach" alternative will greedily match all of the whitespace after foreach , and then fail when it doesn't find an opening ( . It will then backtrack, one whitespace-character at a time, and try to match ( at the previous position until it confirms that it cannot match that line. Using an atomic group (?>\\s*)\\( will cause the regex engine to not backtrack through \\s* if it matches, allowing the regex to fail more quickly.

Be careful when using them though, as they can cause unintended failures when used at the wrong place (for instance, '(?>,*); will never match anything, due to the greedy .* matching all characters (including ; ), and the atomic grouping (?> ... ) preventing the regex engine from backtracking one character to match the ending ; ).

"Unroll the loop" on some of your alternatives, such as your "comment" alternative (also useful if you plan on adding an alternative for strings).

For example: @"| (?<comment> \\@\\* ( ([^\\*]\\@) | (\\*[^\\@]) | . )* \\*\\@ )"

Could be replaced with @"| (?<comment> @\\* [^*]* (\\*+[^@][^*]*)* \\*+@ )"

The new regex boils down to:

1. @\\* : Find the beginning of a comment @*
2. [^*]* : Read all "normal characters" (anything that's not a * because that could signify the end of the comment)
3. (\\*+[^@][^*]*)* : include any non-terminal * inside the comment
   • (\\*+[^@] : If we find a * , ensure that any string of * s doesn't end in a @
   • [^*]* : Go back to reading all "normal characters"
   • )* : Loop back to the beginning if we find another *
4. \\*+@ : Finally, grab the end of the comment *@ being careful to include any extra *

You can find many more ideas for improving the performance of your regular expressions from Jeffrey Friedl's Mastering Regular Expressions (3rd Edition) .