在markdown中检查单词的快速方法？

Question

I want to scan text for the presence of words from a list of words. This would be straightforward if the text were unformatted, but it is markdown-formatted. At the moment, I'm accomplishing this with regex:

import re

text = 'A long text string with **markdown** formatting.'
words = ['markdown', 'markup', 'marksideways']
found_words = []

for word in words:
    word_pattern = re.compile(r'(^|[ \*_])' + word + r'($|[ \*_.!?])', (re.I | re.M))
    match = word_pattern.search(text)
    if match:
        found_words.append(word)

I'm working with a very long list of words (a sort of denylist) and very large candidate texts, so speed is important to me. Is this a relatively efficient and speedy way to do this? Is there a better approach?

Answer 1

Have you considered stripping leading and trailing asterisks?

import re

from timeit import default_timer as timer


text = 'A long text string with **markdown** formatting.'
words = ['markdown', 'markup', 'marksideways']

def regexpCheck(words, text, n):
    found_words = []

    start = timer()
    for i in range(n):
        for word in words:
            word_pattern = re.compile(r'(^|[ \*_])' + word + r'($|[ \*_.!?])', (re.I | re.M))
            match = word_pattern.search(text)
            if match:
                found_words.append(word)

    end = timer()
    return (end - start)


def stripCheck(words, text, n):
    found_words = []

    start = timer()
    for i in range(n):
        for word in text.split():
            candidate = word.strip('*')
            if candidate in words:
                found_words.append(candidate)
    end = timer()

    return (end - start)


n = 10000
print(stripCheck(words, text, n))
print(regexpCheck(words, text, n))

On my run, it's about an order of magnitude faster:

0.010649851000000002
0.12086547399999999