Python：如何确定字符串中是否存在单词列表

Question

给定一个列表["one", "two", "three"] ，如何确定每个单词是否存在于指定的字符串中？

单词列表很短（在我的例子中不到 20 个单词），但要搜索的字符串非常大（每次运行 400,000 个字符串）

我当前的实现使用re来查找匹配项，但我不确定这是否是最好的方法。

import re
word_list = ["one", "two", "three"]
regex_string = "(?<=\W)(%s)(?=\W)" % "|".join(word_list)

finder = re.compile(regex_string)
string_to_be_searched = "one two three"

results = finder.findall(" %s " % string_to_be_searched)
result_set = set(results)
for word in word_list:
    if word in result_set:
        print("%s in string" % word)

我的解决方案中的问题：

1. 它会搜索到字符串的末尾，尽管单词可能出现在字符串的前半部分
2. 为了克服lookahead assertion的限制（不知道如何表达“当前匹配之前的字符应该是非单词字符，或者字符串的开头”），我在字符串前后添加了额外的空格需要搜索。
3. 前瞻断言引入的其他性能问题？

可能的更简单的实现：

1. 只需遍历单词列表并if word in string_to_be_searched执行if word in string_to_be_searched 。 但是如果你要找“三人行”，它就不能处理“三人行”
2. 使用一个正则表达式搜索一个词。 我仍然不确定性能以及多次搜索字符串的潜力。

更新：

我已经接受了 Aaron Hall 的回答https://stackoverflow.com/a/21718896/683321，因为根据 Peter Gibson 的基准https://stackoverflow.com/a/21742190/683321，这个简单版本的性能最好。 如果您对这个问题感兴趣，可以阅读所有答案并获得更好的视图。

实际上我忘了在我原来的问题中提到另一个约束。 单词可以是短语，例如： word_list = ["one day", "second day"] 。 也许我应该问另一个问题。

Answer 1

Peter Gibson（下文）发现此函数是此处答案中性能最高的。 这对于可能保存在内存中的数据集很有用（因为它从要搜索的字符串中创建了一个单词列表，然后是一组这些单词）：

def words_in_string(word_list, a_string):
    return set(word_list).intersection(a_string.split())

用法：

my_word_list = ['one', 'two', 'three']
a_string = 'one two three'
if words_in_string(my_word_list, a_string):
    print('One or more words found!')

其中打印One or words found! 到标准输出。

它确实返回找到的实际单词：

for word in words_in_string(my_word_list, a_string):
    print(word)

打印出来：

three
two
one

---

对于如此大的数据，您无法将其保存在内存中，此答案中给出的解决方案将非常高效。

Answer 2

为了满足我自己的好奇心，我对发布的解决方案进行了计时。 结果如下：

TESTING: words_in_str_peter_gibson          0.207071995735
TESTING: words_in_str_devnull               0.55300579071
TESTING: words_in_str_perreal               0.159866499901
TESTING: words_in_str_mie                   Test #1 invalid result: None
TESTING: words_in_str_adsmith               0.11831510067
TESTING: words_in_str_gnibbler              0.175446796417
TESTING: words_in_string_aaron_hall         0.0834425926208
TESTING: words_in_string_aaron_hall2        0.0266295194626
TESTING: words_in_str_john_pirie            <does not complete>

有趣的是@AaronHall 的解决方案

def words_in_string(word_list, a_string):
    return set(a_list).intersection(a_string.split())

这是最快的，也是最短的之一！ 请注意，它不处理单词旁边的标点符号，但从问题中不清楚这是否是一项要求。 @MIE 和@user3 也建议了此解决方案。

我没有看很长时间为什么两个解决方案不起作用。 如果这是我的错误，请道歉。 这是测试的代码，欢迎评论和更正

from __future__ import print_function
import re
import string
import random
words = ['one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten']

def random_words(length):
    letters = ''.join(set(string.ascii_lowercase) - set(''.join(words))) + ' '
    return ''.join(random.choice(letters) for i in range(int(length)))

LENGTH = 400000
RANDOM_STR = random_words(LENGTH/100) * 100
TESTS = (
    (RANDOM_STR + ' one two three', (
        ['one', 'two', 'three'],
        set(['one', 'two', 'three']),
        False,
        [True] * 3 + [False] * 7,
        {'one': True, 'two': True, 'three': True, 'four': False, 'five': False, 'six': False,
            'seven': False, 'eight': False, 'nine': False, 'ten':False}
        )),

    (RANDOM_STR + ' one two three four five six seven eight nine ten', (
        ['one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten'],
        set(['one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine', 'ten']),
        True,
        [True] * 10,
        {'one': True, 'two': True, 'three': True, 'four': True, 'five': True, 'six': True,
            'seven': True, 'eight': True, 'nine': True, 'ten':True}
        )),

    ('one two three ' + RANDOM_STR, (
        ['one', 'two', 'three'],
        set(['one', 'two', 'three']),
        False,
        [True] * 3 + [False] * 7,
        {'one': True, 'two': True, 'three': True, 'four': False, 'five': False, 'six': False,
            'seven': False, 'eight': False, 'nine': False, 'ten':False}
        )),

    (RANDOM_STR, (
        [],
        set(),
        False,
        [False] * 10,
        {'one': False, 'two': False, 'three': False, 'four': False, 'five': False, 'six': False,
            'seven': False, 'eight': False, 'nine': False, 'ten':False}
        )),

    (RANDOM_STR + ' one two three ' + RANDOM_STR, (
        ['one', 'two', 'three'],
        set(['one', 'two', 'three']),
        False,
        [True] * 3 + [False] * 7,
        {'one': True, 'two': True, 'three': True, 'four': False, 'five': False, 'six': False,
            'seven': False, 'eight': False, 'nine': False, 'ten':False}
        )),

    ('one ' + RANDOM_STR + ' two ' + RANDOM_STR + ' three', (
        ['one', 'two', 'three'],
        set(['one', 'two', 'three']),
        False,
        [True] * 3 + [False] * 7,
        {'one': True, 'two': True, 'three': True, 'four': False, 'five': False, 'six': False,
            'seven': False, 'eight': False, 'nine': False, 'ten':False}
        )),

    ('one ' + RANDOM_STR + ' two ' + RANDOM_STR + ' threesome', (
        ['one', 'two'],
        set(['one', 'two']),
        False,
        [True] * 2 + [False] * 8,
        {'one': True, 'two': True, 'three': False, 'four': False, 'five': False, 'six': False,
            'seven': False, 'eight': False, 'nine': False, 'ten':False}
        )),

    )

def words_in_str_peter_gibson(words, s):
    words = words[:]
    found = []
    for match in re.finditer('\w+', s):
        word = match.group()
        if word in words:
            found.append(word)
            words.remove(word)
            if len(words) == 0: break
    return found

def words_in_str_devnull(word_list, inp_str1):
    return dict((word, bool(re.search(r'\b{}\b'.format(re.escape(word)), inp_str1))) for word in word_list)


def words_in_str_perreal(wl, s):
    i, swl, strwords = 0, sorted(wl), sorted(s.split())
    for w in swl:
        while strwords[i] < w:  
            i += 1
            if i >= len(strwords): return False
        if w != strwords[i]: return False
    return True

def words_in_str_mie(search_list, string):
    lower_string=string.lower()
    if ' ' in lower_string:
        result=filter(lambda x:' '+x.lower()+' ' in lower_string,search_list)
        substr=lower_string[:lower_string.find(' ')]
        if substr in search_list and substr not in result:
            result+=substr
        substr=lower_string[lower_string.rfind(' ')+1:]
        if substr in search_list and substr not in result:
            result+=substr
    else:
        if lower_string in search_list:
            result=[lower_string]

def words_in_str_john_pirie(word_list, to_be_searched):
    for word in word_list:
        found = False
        while not found:
            offset = 0
            # Regex is expensive; use find
            index = to_be_searched.find(word, offset)
            if index < 0:
                # Not found
                break
            if index > 0 and to_be_searched[index - 1] != " ":
                # Found, but substring of a larger word; search rest of string beyond
                offset = index + len(word)
                continue
            if index + len(word) < len(to_be_searched) \
                    and to_be_searched[index + len(word)] != " ":
                # Found, but substring of larger word; search rest of string beyond
                offset = index + len(word)
                continue
            # Found exact word match
            found = True    
    return found

def words_in_str_gnibbler(words, string_to_be_searched):
    word_set = set(words)
    found = []
    for match in re.finditer(r"\w+", string_to_be_searched):
        w = match.group()
        if w in word_set:
             word_set.remove(w)
             found.append(w)
    return found

def words_in_str_adsmith(search_list, big_long_string):
    counter = 0
    for word in big_long_string.split(" "):
        if word in search_list: counter += 1
        if counter == len(search_list): return True
    return False

def words_in_string_aaron_hall(word_list, a_string):
    def words_in_string(word_list, a_string):
        '''return iterator of words in string as they are found'''
        word_set = set(word_list)
        pattern = r'\b({0})\b'.format('|'.join(word_list))
        for found_word in re.finditer(pattern, a_string):
            word = found_word.group(0)
            if word in word_set:
                word_set.discard(word)
                yield word
                if not word_set:
                    raise StopIteration
    return list(words_in_string(word_list, a_string))

def words_in_string_aaron_hall2(word_list, a_string):
    return set(word_list).intersection(a_string.split())

ALGORITHMS = (
        words_in_str_peter_gibson,
        words_in_str_devnull,
        words_in_str_perreal,
        words_in_str_mie,
        words_in_str_adsmith,
        words_in_str_gnibbler,
        words_in_string_aaron_hall,
        words_in_string_aaron_hall2,
        words_in_str_john_pirie,
        )

def test(alg):
    for i, (s, possible_results) in enumerate(TESTS):
        result = alg(words, s)
        assert result in possible_results, \
            'Test #%d invalid result: %s ' % (i+1, repr(result))

COUNT = 10
if __name__ == '__main__':
    import timeit
    for alg in ALGORITHMS:
        print('TESTING:', alg.__name__, end='\t\t')
        try:
            print(timeit.timeit(lambda: test(alg), number=COUNT)/COUNT)
        except Exception as e:
            print(e)

Answer 3

简单的方法：

filter(lambda x:x in string,search_list)

如果您希望搜索忽略字符的大小写，您可以这样做：

lower_string=string.lower()
filter(lambda x:x.lower() in lower_string,search_list)

如果您想忽略属于较大单词的单词，例如三合一：

lower_string=string.lower()
result=[]
if ' ' in lower_string:
    result=filter(lambda x:' '+x.lower()+' ' in lower_string,search_list)
    substr=lower_string[:lower_string.find(' ')]
    if substr in search_list and substr not in result:
        result+=[substr]
    substr=lower_string[lower_string.rfind(' ')+1:]
    if substr in search_list and substr not in result:
        result+=[substr]
else:
    if lower_string in search_list:
        result=[lower_string]

---

如果需要性能：

 arr=string.split(' ') result=list(set(arr).intersection(set(search_list)))

编辑：在一个包含 400,000 个单词的字符串中搜索 1,000 个单词的示例中，此方法是最快的，但如果我们将字符串增加到 4,000,000，则前一种方法更快。

---

如果字符串太长，您应该进行低级搜索并避免将其转换为列表：

 def safe_remove(arr,elem): try: arr.remove(elem) except: pass not_found=search_list[:] i=string.find(' ') j=string.find(' ',i+1) safe_remove(not_found,string[:i]) while j!=-1: safe_remove(not_found,string[i+1:j]) i,j=j,string.find(' ',j+1) safe_remove(not_found,string[i+1:])

not_found列表包含未找到的单词，您可以轻松获取找到的列表，一种方法是list(set(search_list)-set(not_found))

编辑：最后一种方法似乎是最慢的。

Answer 4

def words_in_str(s, wl):
    i, swl, strwords = 0, sorted(wl), sorted(s.split())
    for w in swl:
        while strwords[i] < w:  
            i += 1
            if i >= len(strwords): return False
        if w != strwords[i]: return False
    return True

Answer 5

你可以试试这个：

list(set(s.split()).intersection(set(w)))

它仅从您的单词列表中返回匹配的单词。 如果没有匹配的单词，它将返回空列表。

Answer 6

如果您的字符串很长而您的搜索列表很短，请执行以下操作：

def search_string(big_long_string,search_list)
    counter = 0
    for word in big_long_string.split(" "):
        if word in search_list: counter += 1
        if counter == len(search_list): return True
    return False

Answer 7

您可以使用单词边界：

>>> import re
>>> word_list = ["one", "two", "three"]
>>> inp_str = "This line not only contains one and two, but also three"
>>> if all(re.search(r'\b{}\b'.format(re.escape(word)), inp_str) for word in word_list):
...   print "Found all words in the list"
...
Found all words in the list
>>> inp_str = "This line not only contains one and two, but also threesome"
>>> if all(re.search(r'\b{}\b'.format(re.escape(word)), inp_str) for word in word_list):
...   print "Found all words in the list"
...
>>> inp_str = "This line not only contains one and two, but also four"
>>> if all(re.search(r'\b{}\b'.format(re.escape(word)), inp_str) for word in word_list):
...   print "Found all words in the list"
...
>>>

---

编辑：如您的评论所示，您似乎正在寻找字典：

>>> dict((word, bool(re.search(r'\b{}\b'.format(re.escape(word)), inp_str1))) for word in word_list)
{'three': True, 'two': True, 'one': True}
>>> dict((word, bool(re.search(r'\b{}\b'.format(re.escape(word)), inp_str2))) for word in word_list)
{'three': False, 'two': True, 'one': True}
>>> dict((word, bool(re.search(r'\b{}\b'.format(re.escape(word)), inp_str3))) for word in word_list)
{'three': False, 'two': True, 'one': True}

Answer 8

如果顺序不太重要，可以使用这种方法

word_set = {"one", "two", "three"}
string_to_be_searched = "one two three"

for w in string_to_be_searched.split():
    if w in word_set:
         print("%s in string" % w)
         word_set.remove(w)

.split()创建一个列表，这对于您的 400k 字串可能是一个问题。 但是如果你有足够的内存，你就完成了。

当然可以修改 for 循环以避免创建整个列表。 re.finditer或使用str.find的生成器是显而易见的选择

import re
word_set = {"one", "two", "three"}
string_to_be_searched = "one two three"

for match in re.finditer(r"\w+", string_to_be_searched):
    w = match.group()
    if w in word_set:
         print("%s in string" % w)
         word_set.remove(w)

Answer 9

鉴于你的评论

我实际上并不是在寻找单个 bool 值，而是在寻找将单词映射到 bool 的字典。 此外，我可能需要运行一些测试并查看多次运行 re.search 并运行一次 re.findall 的性能。 – 耶格尔

我会提出以下建议

import re
words = ['one', 'two', 'three']

def words_in_str(words, s):
    words = words[:]
    found = []
    for match in re.finditer('\w+', s):
        word = match.group()
        if word in words:
            found.append(word)
            words.remove(word)
            if len(words) == 0: break
    return found

assert words_in_str(words, 'three two one') == ['three', 'two', 'one']
assert words_in_str(words, 'one two. threesome') == ['one', 'two']
assert words_in_str(words, 'nothing of interest here one1') == []

这将返回按顺序找到的单词列表，但您可以轻松修改它以根据需要返回dict{word:bool} 。

好处：

• 找到所有单词后停止搜索输入字符串
• 一旦找到候选词，就删除它

Answer 10

这是一个简单的生成器，它更适合大字符串或文件，因为我在下面的部分对其进行了调整。

请注意，这应该非常快，但只要字符串继续而不击中所有单词，它就会继续。 这在 Peter Gibson 的基准测试中排名第二： Python：如何确定字符串中是否存在单词列表

有关较短字符串的更快解决方案，请参阅我的其他答案： Python：如何确定字符串中是否存在单词列表

---

原答案

import re

def words_in_string(word_list, a_string):
    '''return iterator of words in string as they are found'''
    word_set = set(word_list)
    pattern = r'\b({0})\b'.format('|'.join(word_list))
    for found_word in re.finditer(pattern, a_string):
        word = found_word.group(0)
        if word in word_set:
            word_set.discard(word)
            yield word
            if not word_set: # then we've found all words
                # break out of generator, closing file
                raise StopIteration 

它在找到单词时遍历字符串，在找到所有单词后或到达字符串末尾时放弃搜索。

用法：

word_list = ['word', 'foo', 'bar']
a_string = 'A very pleasant word to you.'
for word in words_in_string(word_list, a_string):
    print word

word

---

编辑：适应与大文件一起使用：

感谢 Peter Gibson 发现这是第二快的方法。 我为解决方案感到非常自豪。 由于最好的用例是通过一个巨大的文本流，让我在这里调整上面的函数来处理一个文件。 请注意，如果换行符上的单词被破​​坏，这不会捕获它们，但这里的任何其他方法也不会。

import re

def words_in_file(word_list, a_file_path):
    '''
    return a memory friendly iterator of words as they are found
    in a file.
    '''
    word_set = set(word_list)
    pattern = r'\b({0})\b'.format('|'.join(word_list))
    with open(a_file_path, 'rU') as a_file:
        for line in a_file:
            for found_word in re.finditer(pattern, line):
                word = found_word.group(0)
                if word in word_set:
                    word_set.discard(word)
                    yield word
                    if not word_set: # then we've found all words
                        # break out of generator, closing file
                        raise StopIteration

为了演示，让我们写一些数据：

file_path = '/temp/temp/foo.txt'
with open(file_path, 'w') as f:
    f.write('this\nis\nimportant\ndata')

和用法：

word_list = ['this', 'is', 'important']
iterator = words_in_file(word_list, file_path)

我们现在有一个迭代器，如果我们用一个列表来消费它：

list(iterator)

它返回：

['this', 'is', 'important']