如何使用正则表达式降低计算成本 function

Question

我正在尝试使用regex来提取包含特定单词的句子，以及它们之前和之后的句子。 我的代码有效，但每个 txt 需要 20 秒，我有大约一百万个 txt 文件。 是否有可能在更短的时间内获得相同的结果？ 也欢迎任何其他相关建议。 谢谢！

我目前的想法是先提取包含这些目标词的段落，然后使用nltk对目标段落进行标记化，提取目标句子和前后句子。

这是我的演示：

import re, nltk

txt = '''There is widespread agreement that screening for breast cancer, when combined with appropriate follow-up, will reduce mortality from the disease. How we measure response to treatment is called the 5-year survival rate, or the percentage of people who live 5 years after being diagnosed with cancer. According to information provided by National Cancer Institute, Cancer stage at diagnosis, which refers to extent of a cancer in the body, determines treatment options and has a strong influence on the length of survival. In general, if the cancer is found only in the part of the body where it started it is localized (sometimes referred to as stage 1). If it has spread to a different part of the body, the stage is regional or distant . The earlier female breast cancer is caught, the better chance a person has of surviving five years after being diagnosed. For female breast cancer, 62.5 are diagnosed at the local stage. The 5-year survival for localized female breast cancer is 98.8 . It decreases from 98.8 to 85.5 after the cancer has spread to the lymph nodes (stage 2), and to 27.4 
(stage 4) after it has spread to other organs such as the lung, liver or brain. A major problem with current detection methods is that studies have shown that mammography does not detect 10 -20 of breast cancers that are detected by physical exam alone, which may be attributed to a falsely negative mammogram. 

Breast cancer screening is generally recommended as a routine part of preventive healthcare for women over the age of 20 (approximately 90 million in the United States). Besides skin cancer, breast cancer is the most commonly diagnosed cancer among U.S. women. For these women, the American Cancer Society (ACS) has published guidelines for breast cancer screening including: (i) monthly breast self-examinations for all women over the age of 20; (ii) a clinical breast exam (CBE) every three years for women in their 20s and 30s; (iii) a baseline mammogram for women by the age of 40; and (iv) an annual mammogram for women age 40 or older (according to the American College of Radiology). Unfortunately, the U.S. Preventive Task Force Guidelines have stirred confusion by recommending biennial screening mammography for women ages 50-74. 

Each year, approximately eight million women in the United States require diagnostic testing for breast cancer due to a physical symptom, such as a palpable lesion, pain or nipple discharge, discovered through self or physical examination (approximately seven million) or a non-palpable lesion detected by screening x-ray mammography
(approximately one million). Once a physician has identified a suspicious lesion in a woman's breast, the physician may recommend further diagnostic procedures, including a diagnostic x-ray mammography, an ultrasound study, a magnetic resonance imaging procedure, or a minimally invasive procedure such as fine needle aspiration or large core needle biopsy. In each case, the potential benefits of additional diagnostic testing must be balanced against the costs, risks and discomfort to the patient associated with undergoing the additional procedures. 
'''

target_words = ['risks', 'discomfort', 'surviving', 'risking', 'risks', 'risky']
pattern = r'.*\b(?='+'|'.join(target_words) + r')\b.*'
target_paras = re.findall(pattern, txt, re.IGNORECASE)

# Function to extract sentences containing any target word and its neighbor sentences
def UncertaintySentences (paragraph):
    sent_token = nltk.tokenize.sent_tokenize(paragraph)
    keepsents = []

    for i, sentence in enumerate(sent_token):
        # sentences contain any target word
        if re.search(pattern, sentence, re.IGNORECASE) != None:
            try:
                if i==0: # first sentence in a para, keep it and the one next to it
                    keepsents.extend([sent_token[i], sent_token[i+1]])
                elif i!=len(sent_token)-1: # sentence in the middle, keep it ant the ones before and next to it
                    keepsents.extend([sent_token[i-1], sent_token[i], sent_token[i+1]])
                else: # last sentence, keep it and the one before it
                    keepsents.extend([sent_token[i-1], sent_token[i]])
            except: # para with only one sentence
                keepsents = sent_token

    # drop duplicate sentences 
    del_dup = [] 
    [del_dup.append(x) for x in keepsents if x not in del_dup] 
    return(del_dup) 

for para in target_paras:
    uncertn_sents = UncertaintySentences(para)
    print(uncertn_sents)

Answer 1

以下是优化此代码的一些想法：

1. 可以将 target_words 列表转换为集合，使 in 操作更高效。
2. 模式变量可以使用 re.compile 进行预编译，从而使后续对 re.findall 和 re.search 的调用更快。
3. del_dup 列表理解可以替换为 set() 调用以更有效地删除重复项。
4. 也许将 sent_token = nltk.tokenize.sent_tokenize(paragraph) 移出 UncertaintySentences function 的循环，以便每个段落仅执行一次标记化操作。

Answer 2

原始正则表达式的最终速度在很大程度上取决于您正在检查的数据。

您的正则表达式有问题：

r'.*\b(?='+'|'.join(target_words) + r')\b.*'

如果没有关键字的段落很多/很大，那么搜索过程会很慢。

为什么会这样？

因为您的正则表达式以.*开头

您的正则表达式匹配整个段落并开始逐个回溯字符并尝试匹配关键字。 如果根本没有关键字，则回溯过程到达段落的开头。

然后，它再前进一个字符并再次重复整个过程（它到达字符串末尾，回溯到 position 1），然后前进到 position 2 并再次重复一切......

您可以使用此正则表达式调试器更好地查看此过程：

https://regex101.com/r/boZLQU/1/调试器

优化

只需将^添加到您的正则表达式中，如下所示：

r'^.*\b(?='+'|'.join(target_words) + r')\b.*'

这样你只需执行一次回溯过程而不是 N （其中 N 是段落中的字符数）