在C＃中针对10,000个文件优化1000个正则表达式搜索

Question

Given a file that has around 1000 individual regex searches that must be applied across 10,000 other files or so, I'm looking for a nice way to make this run in less than a day's time. No search and replacing going on, just straight match checking. I can manually go through and combine many of these, but I'm wondering if there is an automated tool to do this and perhaps eveng get it down to a single search.

Also wondering if this hasn't been asked 1000 times as well, but my google-foo is failing me here.

Edit: Using an external tool is a possibilitiy, but this has to be run through a script where those 1000 or so searches as passed though and all the results are prettied up into a nice report. I have the side of things done in C#, but it is painfully slow, hence the combine idea or other ways of making it go faster. BTW, I do have it threaded already as well.

Answer 1

First place for improvement is to make sure your loops are nested in the right order.

Loop over the files, and for each file, try all 1000 patterns before moving to the next file. That way you only have to open each file once, not 1000 times.

Second idea is to use a parser generation like flex+yacc+bison to precompile a single DFA (deterministic finite automaton) that covers all the patterns. In the same way that matching against an entire dictionary can use a trie, matching against a list of patterns can generally be done using a state machine with a lot less computation than matching each pattern separately (basically: where and how one pattern fails to match contains information on what patterns might fit in that region)

Answer 2

Found something interesting:

Here is a Regex combiner that runs a Perl script to take all the strings and put them into the One String to rule them all.

Of course, the string that was output from my source strings was roughly 7000 characters in length and that caused C#'s Regex implementaion to blow up spectacularly after roughly 1024 characters - because 1024 characters is all any sane person would need. 2^10 is very magical here.

So who knows if it ran faster after combining because the combination failed too :)

Edit: Hmm, changed a couple of minor things within the search string (removed \\ on quotes) and it runs. Now, time to see if the results of the One String match the results of the 1000 strings. It does appear to run x10 faster though, which is very nice!

Answer 3

I ended up taking a big step back, realizing that 99% of the searches could be easily converted into straight string searches with just a little bit of work, and then using a simple hashset search per word. The remaining searches that needed GREP I left be and it ran plenty fast. KISS.