[英]Transformers pretrained model with dropout setting
I'm trying to use transformer's huggingface pretrained model bert-base-uncased
, but I want to increace dropout.我正在尝试使用 transformer 的 huggingface pretrained model
bert-base-uncased
,但我想增加辍学率。 There isn't any mention to this in from_pretrained
method, but colab ran the object instantiation below without any problem. from_pretrained
方法中没有提到这一点,但 colab 运行下面的 object 实例化没有任何问题。 I saw these dropout parameters in classtransformers.BertConfig
documentation.我在
classtransformers.BertConfig
文档中看到了这些丢失参数。
Am I using bert-base-uncased AND changing dropout in the correct way?我是否以正确的方式使用 bert-base-uncased 和更改 dropout?
model = BertForSequenceClassification.from_pretrained(
pretrained_model_name_or_path='bert-base-uncased',
num_labels=2,
output_attentions = False,
output_hidden_states = False,
attention_probs_dropout_prob=0.5,
hidden_dropout_prob=0.5
)
As Elidor00 already said it, your assumption is correct.正如Elidor00已经说过的,您的假设是正确的。 Similarly I would suggest using a separated Config because it is easier to export and less prone to cause errors.
同样,我会建议使用单独的 Config,因为它更容易导出并且更不容易出错。 Additionally someone in the comments ask how to use it via
from_pretrained
:此外,评论中有人询问如何通过
from_pretrained
使用它:
from transformers import BertModel, AutoConfig
configuration = AutoConfig.from_pretrained('bert-base-uncased')
configuration.hidden_dropout_prob = 0.5
configuration.attention_probs_dropout_prob = 0.5
bert_model = BertModel.from_pretrained(pretrained_model_name_or_path = 'bert-base-uncased',
config = configuration)
Yes this is correct, but note that there are two dropout parameters and that you are using a specific Bert model, that is BertForSequenceClassification
.是的,这是正确的,但请注意,有两个
BertForSequenceClassification
参数,并且您使用的是特定的 Bert 模型,即BertForSequenceClassification
。
Also as suggested by the documentation you could first define the configuration and then the way in the following way:同样按照文档的建议,您可以先定义配置,然后按以下方式定义:
from transformers import BertModel, BertConfig
# Initializing a BERT bert-base-uncased style configuration
configuration = BertConfig()
# Initializing a model from the bert-base-uncased style configuration
model = BertModel(configuration)
# Accessing the model configuration
configuration = model.config
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.