如何从 BERT model 获得词嵌入的余弦相似度

Question

I was interesting in how to get the similarity of word embedding in different sentences from BERT model (actually, that means words have different meanings in different scenarios).

For example:

sent1 = 'I like living in New York.'
sent2 = 'New York is a prosperous city.'

I want to get the cos(New York, New York)'s value from sent1 and sent2, even if the phrase 'New York' is same, but it appears in different sentence. I got some intuition from https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/2

But I still do not know which layer's embedding I need to extract and how to caculate the cos similarity for my above example.

Thanks in advance for any suggestions!

Answer 1

Okay let's do this.

First you need to understand that BERT has 13 layers. The first layer is basically just the embedding layer that BERT gets passed during the initial training. You can use it but probably don't want to since that's essentially a static embedding and you're after a dynamic embedding. For simplicity I'm going to only use the last hidden layer of BERT.

Here you're using two words: "New" and "York". You could treat this as one during preprocessing and combine it into "New-York" or something if you really wanted. In this case I'm going to treat it as two separate words and average the embedding that BERT produces.

This can be described in a few steps:

1. Tokenize the inputs
2. Determine where the tokenizer has word_ids for New and York (suuuuper important)
3. Pass through BERT
4. Average
5. Cosine similarity

First, what you need to import: from transformers import AutoTokenizer, AutoModel

Now we can create our tokenizer and our model:

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
model =

EDIT: hit enter too soon. Adding example