[英]How to access a particular layer of Huggingface's pre-trained BERT model?
For experimentation purposes, I need to access an Embedding layer of the encoder.出于实验目的,我需要访问编码器的嵌入层。 That is, assuming Tensorflow implementation, the layer defined as tf.keras.layers.Embedding(...).
即假设Tensorflow实现,层定义为tf.keras.layers.Embedding(...)。
For example, what is a way to set 'embeddings_regularizer=' argument of the Embedding() layer in the encoder part of the transformer?例如,在转换器的编码器部分中设置 Embedding() 层的“embeddings_regularizer=”参数的方法是什么?
You can iterate over the BERT model in the same way as any other model, like so:您可以像任何其他 model 一样迭代 BERT model,如下所示:
for layer in model.layers:
if isinstance(layer ,tf.keras.layers.Embedding):
layer.embeddings_regularizer = argument
isinstance
checks the type of the layer, so really you can put any layer type here and change what you need. isinstance
检查图层的类型,因此您实际上可以在此处放置任何图层类型并更改您需要的内容。
I haven't checked specifically whether embeddings_regularizer
is available, however if you want to see what methods are available to that particular layer, run a debugger and call dir(layer)
inside the above function.我没有具体检查
embeddings_regularizer
是否可用,但是如果你想查看该特定层可用的方法,请运行调试器并在上述 function 中调用dir(layer)
。
Updated question更新的问题
The TFBertForSequenceClassification model has 3 layers: TFBertForSequenceClassification model 有 3 层:
>>> model.summary()
Model: "tf_bert_for_sequence_classification"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
bert (TFBertMainLayer) multiple 108310272
_________________________________________________________________
dropout_37 (Dropout) multiple 0
_________________________________________________________________
classifier (Dense) multiple 1538
=================================================================
Total params: 108,311,810
Trainable params: 108,311,810
Non-trainable params: 0
Similarly, calling model.layers
gives:同样,调用
model.layers
给出:
[<transformers.models.bert.modeling_tf_bert.TFBertMainLayer at 0x7efda85595d0>,
<tensorflow.python.keras.layers.core.Dropout at 0x7efd6000ae10>,
<tensorflow.python.keras.layers.core.Dense at 0x7efd6000afd0>]
We can access the layers inside TFBertMainLayer
:我们可以访问
TFBertMainLayer
中的层:
>>> model.layers[0]._layers
[<transformers.models.bert.modeling_tf_bert.TFBertEmbeddings at 0x7efda8080f90>,
<transformers.models.bert.modeling_tf_bert.TFBertEncoder at 0x7efda855ced0>,
<transformers.models.bert.modeling_tf_bert.TFBertPooler at 0x7efda84f0450>,
DictWrapper({'name': 'bert'})]
So from the above we can access the TFBertEmbeddings layer by:所以从上面我们可以通过以下方式访问 TFBertEmbeddings 层:
model.layers[0].embeddings
OR
model.layers[0]._layers[0]
If you check the documentation (search for the "TFBertEmbeddings" class) you can see that this inherits a standard tf.keras.layers.Layer
which means you have access to all the normal regularizer methods, so you should be able to call something like:如果您查看文档(搜索“TFBertEmbeddings”类),您可以看到它继承了标准
tf.keras.layers.Layer
这意味着您可以访问所有正常的正则化方法,因此您应该能够调用类似:
from tensorflow.keras import regularizers
model.layers[0].embeddings.activity_regularizer = regularizers.l2(1e-5)
Or whatever argument / regularizer you need to change.或者您需要更改的任何参数/正则化器。 See here for regularizer docs.
有关正则化器文档,请参见此处。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.