使用 word2vec 列出所有包含特定单词和相似单词的句子

Question

I have a table as following:

data = {'text':  ['The scent is nice','I like the smell', 'The smell is awesome', 'I find the scent amazing', 'I love the smell']}

df = pd.DataFrame (data, columns = ['text'])

I want to list all sentences that contain the word "smell"

word = 'smell'
selected_list = []
for i in range(0, len(df)):
    if word in df.iloc[i,0]:
        selected_list.append(df.iloc[i,0])
selected_list

The output that I get is:

['I like the smell', 'The smell is awesome', 'I love the smell']

However, I want to list also sentences that contain a similar word to "smell" such as "scent" and I want to use the pre-trained word2vec of Google and set up a condition, if the similarity is above 0.5 to list the sentence as well. Therefore, the desired output is:

['The scent is nice', 'I like the smell', 'The smell is awesome', 'I find the scent amazing','I love the smell']

How can I add word2vec to the above code so that it scans not only for "smell" but also all similar words?

Answer 1

It sounds like you'll want to compare each word in a candidate text to your query-word, and then see if one (or more) of the most-similar words are over your threshold.

That will require tokenizing the raw texts into words that are suitable for lookup against your set-of-word-vectors, then comparing/sorting the results, then checking them against your threshold.

The core of what you need to do could be the following function, reliant on the word-vectors support in Python library gensim :

def rank_by_similarity(target_word, list_of_words, word_vectors):
    """Return ranked list of (similarity_score, word) of words in
    list_of_words, by similarity to target_word, using the set of
    vectors in word_vectors (a gensim.models.KeyedVectors instance)."""

    sim_pairs = [(word_vectors.similarity(target_word, word), word) 
                 for word in list_of_words]
    sim_pairs.sort(reverse=True)  # put largest similarities first
    return sim_pairs

With use in context:

from gensim.models import KeyedVectors

all_sentences = [
    'That looks nice',
    'The scent is nice',
    'It tastes great', 
    'I like the smell',
    'Wow that\'s hot',
]
query_word = 'smell'
threshold = 0.5

goog_vecs = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)

selected_sentences = []
for sentence in all_sentences:
    tokens = sentence.split()
    ranked_tokens = rank_by_similarity(query_word, tokens, goog_vecs)
    if ranked_tokens[0][0] > threshold:  # if top match similarity > threshold...
        selected_sentences.append(sentence)  # ...consider it a match
print(selected_sentences)