在字符串python中查找匹配的短语和单词

Question

Using python, what would be the most efficient way for one to extract common phrases or words from to given string?

For example,

string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"

Would return:

["a","time","there","was a very","called Jack"] 

How would one go about in doing this efficiently (in my case I would need to do this over thousands of 1000 word documents)?

Answer 1

You can split each string, then intersect the set s.

string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"
set(string1.split()).intersection(set(string2.split()))

Result

set(['a', 'very', 'Jack', 'time', 'was', 'called'])

Note this only matches individual words. You have to be more specific on what you would consider a "phrase". Longest consecutive matching substring? That could get more complicated.

Answer 2

In natural language processing, you usually extract common patterns and sequences from sentences using n-grams . In python, you can use the excellent NLTK module for that.

For counting and finding the most common, you can use collections.Counter .

Here's a example for 2-grams:

from nltk.util import ngrams
from collections import Counter
from itertools import chain

string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"

n = 2
ngrams1= ngrams(string1.split(" "), n)
ngrams2= ngrams(string2.split(" "), n)

counter= Counter(chain(ngrams1,ngrams2))       #count occurrences of each n-gram
print [k[0] for k,v in counter.items() if v>1] #print all ngrams that come up more than once

output:

[('called', 'Jack'), ('was', 'a'), ('a', 'very')]

output with n=3 :

[('was', 'a', 'very')]

output with n=1 (without tuples):

['Jack', 'a', 'was', 'time', 'called', 'very']

Answer 3

This is a classic dynamic programming problem. All you need to do is build a suffix tree for string1 , with words instead of letters (which is the usual formulation). Here is an illustrative example of a suffix tree .

1. Label all nodes in your tree as s1 .
2. Insert all suffixes of string2 one by one.
3. All nodes that the suffixes in step 2 pass through are labeled s2 .
4. Any new nodes created in step 2 are also labeled s2 .
5. In the final tree, path labels of every node labeled both s1 and s2 is a common substring.

This algorithm is succinctly explained in this lecture note .

For two strings of lengths n and m , the suffix tree construction takes O(max(n,m)) , and all the matching substrings (in your case, words or phrases) can be searched in O(#matches) .

Answer 4

A couple of years later, but I tried this way using 'Counter' below:

Input[ ]:

from collections import Counter

string1="once upon a time there was a very large giant called Jack"
string2="a very long time ago was a very brave young man called Jack"
string1 += ' ' + string2
string1 = string1.split()

count = Counter(string1)
tag_count = []
for n, c in count.most_common(10):
    dics = {'tag': n, 'count': c}
    tag_count.append(dics)

Output[ ]:

[{'tag': 'a', 'count': 4},
 {'tag': 'very', 'count': 3},
 {'tag': 'time', 'count': 2},
 {'tag': 'was', 'count': 2},
 {'tag': 'called', 'count': 2},
 {'tag': 'Jack', 'count': 2},
 {'tag': 'once', 'count': 1},
 {'tag': 'upon', 'count': 1},
 {'tag': 'there', 'count': 1},
 {'tag': 'large', 'count': 1}]

Hopefully, it would be useful for someone :)