python-在字符串列表中，查找至少出现在y个条目中的至少具有n个连续标记的所有模式

Question

The task I am trying to accomplish is to write a function that would identify all patterns of at least n tokens that occur in at least y entries when searching within a list of strings.

For example:

list = ["Hello my name is foobar","Hello my favorite food is pizza","Hello my favorite food will never be broccoli","No my name is not barfoo", "Yes my name is foobar"]

Then

function(list, n=3, y=3)
["my name is"]

function(list, n=3, y=2)
["my name is", "my favorite food"]

I would like to use this function with extremely large lists. I was planning to do this the brute force way with multiple nested loops, but this would be extremely slow. I am wondering if there are more efficient ways to do this type of a task.

Answer 1

Here's a quick function to do this. In this function each sentence is broken in n_tokens -grams. Wrapping set() around the ngrams will ensure that only distinct ngrams are included and if an ngram occurs multiple times in a sentence it won't be double-counted later. Using itertools the word_grams from the sentences are combined and Counter will count the number of occurrence for each ngram . Finally, the counts of the gram_occur are evaluated to see what grams occur most frequently. The result is converted into a list of strings that meet your criteria.

from nltk import ngrams
import itertools
from collections import Counter


def count_ngrams(l, n_tokens, min_occur):
    word_grams = [set(ngrams(s.split(), n_tokens)) for s in l]

    gram_occur = Counter(itertools.chain.from_iterable(word_grams))

    return [" ".join([*words]) for (words, n) in gram_occur.items() if n >= min_occur]