返回评论列表和句子列表中的单词列表

Question

I have a long (1.5m) list of sentences and a similarly long list of words that I am looking for within the list of sentences. For example:

list_of_words = ['Turin', 'Milan']
list_of_sents = ['This is a sent about turin.', 'This is a sent about manufacturing.']

I would like to have a function that is able to return those sentences that contain the target word without alphanumeric characters next to them. In other words, only the first of the sentences above should match.

I have developed the function below but it takes too long to parse through each the millions of words and sentences. I was wondering if there is a package or alternative that could mitigate this computational intensity.

def find_target_sents(list_of_words, list_of_sents):
     target_sents = []
     i, j = 0, 0
     word_len = len(list_of_words)
     sent_len = len(list_of_sents)
     for word in list_of_words:
        i += 1
        for sent in list_of_sents:
            j += 1
            print('%s out of %s words and %s out of %s sentences' % (j, word_len , i, sent_len))
            match = re.compile(r'\%s\b' % word, re.I)
            y = match.search(sent)
            if y != None:
                print(sent)
                t = (word, sentence)
                target_sent.append(t)
     print(target_sent)

Answer 1

if you could build a string with all the words to search from list_of_words like (Turin|Milan) , you could do a regex match on:

^.*\b(Turin|Milan)\b.*$

Also, we could avoid both the for loops, as mentioned in this answer .

Answer 2

Could just build sets and use their constant time membership check:

from string import punctuation

def find_target_sents(words, sents):
    # translation table
    table = str.maketrans('', '', punctuation) 
    # hold found sentences by word
    found = {word: [] for word in words} 
    # make unique sets for each sentence and remove punctuation
    parsed = [set(sent.translate(table).split()) for sent in sents]
    # check
    for word in found:
        for sent in parsed:
            if word in sent:
                found[word].append(sent)

Basically this assumes your sentences follow English grammar and are separated with spaces following a chracter or punctionation (assuming another word will follow of course).

It takes each sentence and removes any punctuation from it, then splits on whitespace and turns the result into a set which has constant time, O(1), membership check.

So the sentence: "I, want to go, to Burger King!" ...

... becomes ('I', 'want', 'to', 'go', 'Burger', 'King') ; where only unique elements exist!

Obviously there are issues if you are looking for 'Burger King' but thats technically two words...