使用 huggingface/transformers 预训练 model 与 SentenceTransformer 的句子转换器

Question

. Answers to this question are eligible for a +50 reputation bounty. user2543622 is looking for an .

This page has two scripts

When should one use 1st method shown below vs 2nd? As nli-distilroberta-base-v2 trained specially for finding sentence embedding wont that always be better than the first method?

training_stsbenchmark.py1 -

from sentence_transformers import SentenceTransformer,  LoggingHandler, losses, models, util
#You can specify any huggingface/transformers pre-trained model here, for example, bert-base-uncased, roberta-base, xlm-roberta-base
model_name = sys.argv[1] if len(sys.argv) > 1 else 'distilbert-base-uncased'

# Use Huggingface/transformers model (like BERT, RoBERTa, XLNet, XLM-R) for mapping tokens to embeddings
word_embedding_model = models.Transformer(model_name)

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)

model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

training_stsbenchmark_continue_training.py -

from sentence_transformers import SentenceTransformer, LoggingHandler, losses, util, InputExample
model_name = 'nli-distilroberta-base-v2'
model = SentenceTransformer(model_name)

Answer 1

You are comparing 2 different things:

training_stsbenchmark.py - This example shows how to create a SentenceTransformer model from scratch by using a pre-trained transformer model together with a pooling layer.

In other words, you are creating your own model SentenceTransformer using your own data , therefore fine-tuning.

training_stsbenchmark_continue_training.py - This example shows how to continue training on STS data for a previously created & trained SentenceTransformer model.

In that example, they load a model trained on NLI data.

So, to answer "wont that always be better than the first method?"

It depends on you final results. Try both methods and check for yourself which will deliver better cross-validation results.