如何使用深度学习找到句子相似度？

Question

I am trying to find sentence similarity through word emebeddings and then applying cosine similarity score. Tried CBOW/Skip Gram methods for embedding but did not solve the problem.

I am doing this for product review data. I have two columns:

SNo         Product_Title                                Customer_Review   
 1       101.x battery works well                    I have an Apple phone and it's not that
          with Samsung smart phone                     that great.

 2       112.x battery works well                     I have samsung smart tv and I tell that it's
         with Samsung smart phone                     not wort buying.

 3      112.x battery works well                      This charger works very well with samsung 
        with Samsung smart phone.                      phone. It is fast charging.

The first two reviews are irrelevant as semantic meaning of Product_Title and Customer_Review are completely different.

How can an algorithm find this semantic meaning of sentences and score them.

My Approach:

1. Text pre-processing

2. Train CBOW/Skip gram using Gensim on my data-set

3. Do Sentence level encoding via averaging all word vectors in that sentence

4. Take cosine similarity of product_title and reviews .

Problem: It was not able to find the context from the sentence and hence the result was very poor.

Approch 2:

Used pre-trained BERT without pre-processing sentences. The result was not improving either.

1.Any other approach that would capture the context/semantics of sentences.

2.How can we train BERT on our data-set from scratch without using pre-trained model?

Answer 1

Have you tried the Universal Sentence Encoder (USE) , or the Multilingual Universal Sentence Encoder ?

There's a colab showing how to score sentence pairs for semantic textual similarity with USE on the Semantic Textual Similarity Benchmark (STS-B) and another for multilingual similarity .

Here's a heatmap of pairwise semantic similarity scores from USE on the Google AI blog post Advances in Semantic Textual Similarity . The model was trained on a large amount of web data so it should work well for a wide variety of input data.

Answer 2

Here is a very elaborate tutorial on how to perform sentence similarity analysis using any of the 50+ sentence Embeddings in NLU, like BERT, USE, Electra, and many more! NLU features over 50+ languages and includes multilingual embeddings!
It takes around 5 lines to generate a similarity matrix with NLU and you can use 3 or more Sentence Embeddings at the same time in just 1 line of code, all you need is :

nlu.load('embed_sentence.bert embed_sentence.electra use')

But let's keep it simple and let's say we want to calculate the similarity matrix for every sentence in our Dataframe

You need the following 3 steps :

1. Calculate embeddings

predictions = nlu.load('embed_sentence.bert').predict(your_dataframe)

2. Calculate the similarity matrix

def get_sim_df_total( predictions,e_col, string_to_embed,pipe=pipe):
  # This function calculates the distances between every sentence pair. Creates for ever sentence a new column with the name equal to the sentence it comparse to 
  # put embeddings in matrix
  embed_mat = np.array([x for x in predictions[e_col]])
  # calculate distance between every embedding pair
  sim_mat = cosine_similarity(embed_mat,embed_mat)
  # for i,v in enumerate(sim_mat): predictions[str(i)+'_sim'] = sim_mat[i]
  for i,v in enumerate(sim_mat): 
    s = predictions.iloc[i].document
    predictions[s] = sim_mat[i]

  return predictions 

sim_matrix_df = get_sim_df_total(predictions,'embed_sentence_bert_embeddings', 'How to get started with Machine Learning and Python' )
sim_matrix_df

3. Plot the heatmap of the similarity matrix

non_sim_columns  = ['text','document','Title','embed_sentence_bert_embeddings']

def viz_sim_matrix_first_n(num_sentences=20, sim_df = sim_matrix_df):
  # Plot heatmap for the first num_sentences
  fig, ax = plt.subplots(figsize=(20,14)) 
  sim_df.index = sim_df.document
  sim_columns = list(sim_df.columns)
  for b in non_sim_columns : sim_columns.remove(b)
  # sim_matrix_df[sim_columns]
  ax = sns.heatmap(sim_df.iloc[:num_sentences][sim_columns[:num_sentences]]) 

  ax.axes.set_title(f"Similarity matrix for the first {num_sentences} in the dataset",)

viz_sim_matrix_first_n()

To learn more checkout these links :)

Article : https://medium.com/spark-nlp/easy-sentence-similarity-with-bert-sentence-embeddings-using-john-snow-labs-nlu-ea078deb6ebf

Colab Notebook for sentence similarity Demo with NLU : https://colab.research.google.com/drive/1LtOdtXtRJ3_N8kYywPd5k2AJMCGcgAdN?usp=sharing

NLU Website : http://nlu.johnsnowlabs.com/