Pandas 数据框列的 BERT Word Embedding

Question

I m working on a NLP project using Tamil Universal Dependency dataset. I have preprocessed the data into a data frame, of which columns are tokens and its dependency tags. I would like to perform word embedding using mBERT model. Since the dataset is a pretrained model, it is already tokenized as seen in the attached Data frame. I m not sure how to proceed because, when tokens are converted to token id's are wrongly marked by the tokenizer.

b #List of tokens

Data Frame

Token ID Error

Answer 1

You can find some example code and explanations here: https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958

Important point: the added value of BERT is to generate contextualised embeddings, ie embeddings for longer sequences of text (the context) where the embeddings for each individual word change depending on the surrounding words. When you only want static embeddings for individual words (independent of context), then BERT is not the right tool and its better to use static embeddings like Glove, Word2Vec, FastText. Its well known that BERT does not produce good individual word embeddings.

What makes sense for you depends on your use-case, but the way you have preprocessed your text indicates that you actually want static embeddings.