使用python将文本与大量正则表达式进行比较的最快方法是什么？

Question

I have a list of regular expressions and I would like to match with tweets that as they arive so I can associate them with a specific account. With a small number of rules as above it goes really fast, but as soon as you increase the amount of rules, it becomes slower and slower.

import string, re2, datetime, time, array

rules = [
    [[1],["(?!.*ipiranga).*((?=.*posto)(?=.*petrobras).*|(?=.*petrobras)).*"]],
    [[2],["(?!.*brasil).*((?=.*posto)(?=.*petrobras).*|(?=.*petrobras)).*"]],
]

#cache compile
compilled_rules = []
for rule in rules:
    compilled_scopes.append([[rule[0][0]],[re2.compile(rule[1][0])]])

def get_rules(text):
    new_tweet = string.lower(tweet)
    for rule in compilled_rules:
        ok = 1
        if not re2.search(rule[1][0], new_tweet): ok=0
        print ok

def test():
    t0=datetime.datetime.now()
    i=0
    time.sleep(1)
    while i<1000000:
        get_rules("Acabei de ir no posto petrobras. Moro pertinho do posto brasil")
        i+=1
        t1=datetime.datetime.now()-t0
        print "test"
        print i
        print t1
        print i/t1.seconds

When I have tested with 550 rules, I couldn't do more then 50 reqs/s. Is there a better way for doing this? I need at least 200 reqs/s

EDIT: after tips from Jonathan I could improve about speed 5 times just but nesting a bit my rules. See the code below:

scope_rules = {
    "1": {
        "termo 1" : "^(?!.*brasil)(?=.*petrobras).*",
        "termo 2" : "^(?!.*petrobras)(?=.*ipiranga).*",
        "termo 3" : "^(?!.*petrobras)(?=.*ipiranga).*",
        "termo 4" : "^(?!.*petrobras)(?=.*ipiranga).*",
        },
    "2": {
        "termo 1" : "^(?!.*ipiranga)(?=.*petrobras).*",
        "termo 2" : "^(?!.*petrobras)(?=.*ipiranga).*",
        "termo 3" : "^(?!.*brasil)(?=.*ipiranga).*",
        "termo 4" : "^(?!.*petrobras)(?=.*ipiranga).*",
        }
    }
compilled_rules = {}
for scope,rules in scope_rules.iteritems():
    compilled_rules[scope]={}
    for term,rule in rules.iteritems():
        compilled_rules[scope][term] = re.compile(rule)


def get_rules(text):
    new_tweet = string.lower(text)
    for scope,rules in compilled_rules.iteritems():
        ok = 1
        for term,rule in rules.iteritems():
            if ok==1:
                if re.search(rule, new_tweet):
                    ok=0
                    print "found in scope" + scope + " term:"+ term


def test():
    t0=datetime.datetime.now()
    i=0
    time.sleep(1)
    while i<1000000:
        get_rules("Acabei de ir no posto petrobras. Moro pertinho do posto ipiranga da lagoa")
        i+=1
        t1=datetime.datetime.now()-t0
        print "test"
        print i
        print t1
        print i/t1.seconds

cProfile.run('test()', 'testproof')

Answer 1

Your rules appear to be the culprits here: Because of the two .* , separated by lookaheads, a very high number of permutations has to be checked for a successful match (or to exclude a match). This is further compounded by your using re.search() without anchors. Also, the alternation including the posto part is superfluous - the regex matches whether or not there's any posto in your string, so you might as well drop that completely.

For example, your first rule can be rewritten as

^(?!.*ipiranga)(?=.*petrobras)

without any change in results. You can further optimize it with word boundaries, if you're looking for exact words:

^(?!.*\bipiranga\b)(?=.*\petrobras\b)

---

Some measurements (using RegexBuddy ):

Your first regex, applied to the string Acabei de ir no posto petrobras. Moro pertinho do posto brasil Acabei de ir no posto petrobras. Moro pertinho do posto brasil takes the regex engine about 4700 steps to figure out a match. If I take out the s in petrobras , it takes over 100.000 steps to determine a non-match.

Mine matches in 230 steps (and fails in 260), so you get a 20-400 times speed-up just from constructing the regex correctly.

Answer 2

Going even further I created a Cython extension to evaluate the rules and now it's blazing fast. I can do about 70 requests per second with about 3000 regex rules

regex.pyx

import re2
import string as pystring

cpdef list match_rules(char *pytext, dict compilled_rules):
    cdef int ok, scope, term
    cdef list response = []
    text = pystring.lower(pytext)
    for scope, rules in compilled_rules.iteritems():
        ok = 1
        for term,rule in rules.iteritems():
            if ok==1:
                if re2.search(rule, text):
                    ok=0
                    response.append([scope,term])
    return response

python code

import re2 as re
import datetime, time, cProfile

scope_rules = {1: {1 : "^(?!.*brasil)(?=.*petrobras).*", 2: "^(?!.*petrobras)(?=.*ipiranga).*",3 : "^(?!.*petrobras)(?=.*ipiranga).*",4 : "^(?!.*petrobras)(?=.*ipiranga).*",},2: {1 : "^(?!.*brasil)(?=.*petrobras).*", 2: "^(?!.*petrobras)(?=.*ipiranga).*",3 : "^(?!.*petrobras)(?=.*ipiranga).*",4 : "^(?!.*petrobras)(?=.*ipiranga).*",},}

compilled_rules = {}
for scope,rules in scope_rules.iteritems():
    compilled_rules[scope]={}
    for term,rule in rules.iteritems():
        compilled_rules[scope][term] = re.compile(rule)

def test():
    t0=datetime.datetime.now()
    i=0
    time.sleep(1)
    while i<1000000:
        mregex.match_rules("Acabei de ir no posto petrobras. Moro pertinho do posto brasil",compilled_rules)
        i+=1
        t1=datetime.datetime.now()-t0
        print t1
        print i/t1.seconds

cProfile.run('test()', 'testproof')

Answer 3

Besides optimizing your regex patterns themselves (which will make a huge difference), you can try Google's RE2 - It's supposed to be faster than Python's standard regular expressions module.

It's done in C++, but there's PyRE2 , a Python wrapper for RE2 by Facebook :)

PS Thanks to your question, I found a great read on regex matching!

Answer 4

in addition to @Tim Pietzcker 's recommendation

if you have a lot of rules, it might make sense to try a tiered approach built off of smaller rules grouped by commonalities.

for example, both of your rules above match 'posto' and 'petrobras'. if grouped those regexes into a list together, and then qualified a dispatch to that list, you could avoid running lots of rules that would never apply.

in pseducode....

# a dict of qualifier rules with list of rules 
rules = {
    "+ petrobras" : [
        "- ipiranga"
        "- brasil"
        "? posto"
    ] ,
}

for rule_qualifier in rules:
   if regex( rule_qualifier , text ):
       for rule in rule_qualifier:
           if regex( rule , text ):
               yay