python-正则表达式模式是大量项目的最佳实践？

Question

I'm looking for a pro tip here. I have in a database a list of strings, things like "MD", "PHD", "MR" etc. various salutations. It's several hundred rows and I receive it in a specific order (MD is more important than MR). I also have a series of people objects that I'll be iterating and need a very efficient way of matching. I've tried two and maybe there isn't another method.

My first try is when I receive the list, re.compile each one and put them into a list. Then...

theregexlist = ["MR", "DR", "MRS" ... "MISS", "PHD"] #several hundred
personname = "MR JOEY SMITH" #other examples are similar like "BOBBY DR MD JOE"
for theregex in theregexlist:
    if re.search(theregex, personname):
        do stuffs....
        break #since my list is ordered, I only want the first match

Which does indeed work. I also tried looping the regexlist and building a huge matching regex with capturing parans, re.compile it, and then:

hugeregex = re.compile("(?:(MR)|(MR)|(PHD)| ...  |(DR)|(MD))")
personname = "FRED DR FLINTSTONE"
maybematch = re.search(hugeregex, personname)
if maybematch:
    print (maybematch.group(0))

Is there some kind of map, leverage keys, or iteration function that I'm just not thinking of that would be more efficient? Any and all idea are appreciated! Even if it's "Yup, it's just gonna be slow, try to use timeit to see which is faster", then I can stop searching :) Thank you!

Answer 1

The "big" RegEx with all "particules" (like "MR", "MS", etc.), will be more efficient because it will be compiled only once. And you reduce function calls (which is an optimisation).

If you have special characters inside a particule, you may need to escape them with re.escape .

You can compile the RegEx and get a reference to the search method.

Here is an example:

import re

particules = ["MR", "DR", "MRS", "MISS", "PHD"]

regex = r"\b(?:" + "|".join(map(re.escape, particules)) + r")\b"
search_any_particule = re.compile(regex, flags=re.IGNORECASE).search

personname = "FRED DR FLINTSTONE"

mo = search_any_particule(personname)
if mo:
    print(mo.group())

You get: 'DR'.

EDIT

The best way to make sure you implementation is efficient is to profile it. For that, you can use cProfile library.

For instance:

def find_particule(personname):
    mo = search_any_particule(personname)
    if mo:
        return mo.group()
    return None

import cProfile

cProfile.runctx('for i in range(1000000): find_particule("FRED DR FLINTSTONE")', globals(), locals())

The profiler will give you something like this:

         3000003 function calls in 2.110 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.353    0.353    2.110    2.110 <string>:1(<module>)
  1000000    0.495    0.000    1.757    0.000 python:10(find_particule)
        1    0.000    0.000    2.110    2.110 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
  1000000    0.185    0.000    0.185    0.000 {method 'group' of '_sre.SRE_Match' objects}
  1000000    1.078    0.000    1.078    0.000 {method 'search' of '_sre.SRE_Pattern' objects}