[英]Find matching phrases and words in a string python
使用python,从给定字符串中提取常用短语或单词的最有效方法是什么?
例如,
string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"
会返回:
["a","time","there","was a very","called Jack"]
如何有效地做到这一点(在我的情况下,我需要在数千个 1000 字的文档中做到这一点)?
您可以split
每个字符串,然后intersect
set
s intersect
。
string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"
set(string1.split()).intersection(set(string2.split()))
结果
set(['a', 'very', 'Jack', 'time', 'was', 'called'])
请注意,这仅匹配单个单词。 您必须更具体地了解您认为的“短语”。 最长的连续匹配子串? 那可能会变得更复杂。
在自然语言处理中,您通常使用n-grams
从句子中提取常见的模式和序列。 在 python 中,你可以使用优秀的NLTK
模块。
为了计数和找到最常见的,您可以使用collections.Counter
。
以下是 2 克的示例:
from nltk.util import ngrams
from collections import Counter
from itertools import chain
string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"
n = 2
ngrams1= ngrams(string1.split(" "), n)
ngrams2= ngrams(string2.split(" "), n)
counter= Counter(chain(ngrams1,ngrams2)) #count occurrences of each n-gram
print [k[0] for k,v in counter.items() if v>1] #print all ngrams that come up more than once
输出:
[('called', 'Jack'), ('was', 'a'), ('a', 'very')]
输出n=3
:
[('was', 'a', 'very')]
n=1
输出(没有元组):
['Jack', 'a', 'was', 'time', 'called', 'very']
几年后,但我使用下面的“计数器”尝试了这种方式:
输入[ ]:
from collections import Counter
string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"
string1 += ' ' + string2
string1 = string1.split()
count = Counter(string1)
tag_count = []
for n, c in count.most_common(10):
dics = {'tag': n, 'count': c}
tag_count.append(dics)
输出[ ]:
[{'tag': 'a', 'count': 4},
{'tag': 'very', 'count': 3},
{'tag': 'time', 'count': 2},
{'tag': 'was', 'count': 2},
{'tag': 'called', 'count': 2},
{'tag': 'Jack', 'count': 2},
{'tag': 'once', 'count': 1},
{'tag': 'upon', 'count': 1},
{'tag': 'there', 'count': 1},
{'tag': 'large', 'count': 1}]
希望它对某人有用:)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.