[英]How to take the trainable parameters into a loss function in Tensoflow.Keras
我正在嘗試實現損失 function ,其中需要卷積層中的變量進行計算。 官方文檔給出了一種方法,涉及loss中的變量function:
如果您的損失不是這種情況(例如,如果您的損失引用模型層之一的變量),您可以將損失包裝在零參數 lambda 中。 這些損失沒有作為模型拓撲的一部分進行跟蹤,因為它們不能被序列化。
inputs = tf.keras.Input(shape=(10,))
x = tf.keras.layers.Dense(10)(inputs)
outputs = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs, outputs)
# Weight regularization.
model.add_loss(lambda: tf.reduce_mean(x.kernel))
然而,這只是向 model 添加一個簡單的正則化。 有沒有辦法實現更復雜的正則化器,其中涉及不同層中變量之間的計算? 如果一個可訓練變量也被添加到正則化器中呢?
您可以使用add_loss
API 添加任意復雜的損失函數。 這是一個添加使用兩個不同層權重的損失的示例。
import tensorflow as tf
print('TensorFlow:', tf.__version__)
inp = tf.keras.Input(shape=[10])
x = tf.keras.layers.Dense(16)(inp)
x = tf.keras.layers.Dense(32)(x)
x = tf.keras.layers.Dense(4)(x)
out = tf.keras.layers.Dense(1)(x)
model = tf.keras.Model(inputs=[inp], outputs=[out])
model.summary()
def custom_loss(weight_a, weight_b):
def _custom_loss():
# This can include any arbitrary logic
loss = tf.norm(weight_a) + tf.norm(weight_b)
return loss
return _custom_loss
weight_a = model.layers[2].kernel
weight_b = model.layers[3].kernel
model.add_loss(custom_loss(weight_a, weight_b))
print('\nlosses:', model.losses)
Output:
TensorFlow: 2.3.0-dev20200611
Model: "functional_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 10)] 0
_________________________________________________________________
dense (Dense) (None, 16) 176
_________________________________________________________________
dense_1 (Dense) (None, 32) 544
_________________________________________________________________
dense_2 (Dense) (None, 4) 132
_________________________________________________________________
dense_3 (Dense) (None, 1) 5
=================================================================
Total params: 857
Trainable params: 857
Non-trainable params: 0
_________________________________________________________________
losses: [<tf.Tensor: shape=(), dtype=float32, numpy=7.3701963>]
受@Srihari Humbarwadi 的啟發,我找到了一種實現復雜正則化的方法,包括:
這個想法是構造一個子類 model:
class Pseudo_Model(Model):
def __init__(self, **kwargs):
super(Pseudo_Model, self).__init__(**kwargs)
self.dense1 = Dense(16)
self.dense2 = Dense(4)
self.dense3 = Dense(2)
self.a = tf.Variable(shape=(1,), initial_value=tf.ones(shape=(1,)))
def call(self, inputs, training=True, mask=None):
x = self.dense1(inputs)
x = self.dense2(x)
x = self.dense3(x)
return x
model 通過以下方式構建:
sub_model = Pseudo_Model(name='sub_model')
inputs = Input(shape=(32,))
outputs = sub_model(inputs)
model = Model(inputs, outputs)
model.summary()
model.get_layer('sub_model').summary()
model的結構:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 32)] 0
_________________________________________________________________
sub_model (Pseudo_Model) (None, 2) 607
=================================================================
Total params: 607
Trainable params: 607
Non-trainable params: 0
_________________________________________________________________
Model: "sub_model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dense (Dense) (None, 16) 528
_________________________________________________________________
dense_1 (Dense) (None, 4) 68
_________________________________________________________________
dense_2 (Dense) (None, 2) 10
=================================================================
Total params: 607
Trainable params: 607
Non-trainable params: 0
_________________________________________________________________
然后像@Srihari Humbarwadi 提到的那樣定義損失function,只是添加一個新的可訓練參數a:
def custom_loss(weight_a, weight_b, a):
def _custom_loss():
# This can include any arbitrary logic
loss = a * tf.norm(weight_a) + tf.norm(weight_b)
return loss
return _custom_loss
通過 add_loss() API 將損失添加到 model 中:
a_ = model.get_layer('sub_model').a
weighta = model.get_layer('sub_model').layers[0].kernel
weightb = model.get_layer('sub_model').layers[1].kernel
model.get_layer('sub_model').add_loss(custom_loss(weighta, weightb, a_))
print(model.losses)
#[<tf.Tensor: id=116, shape=(1,), dtype=float32, numpy=array([7.2659254], dtype=float32)>]
然后我創建一個假數據集來測試它:
fake_data = np.random.rand(1000, 32)
fake_labels = np.random.rand(1000, 2)
model.compile(optimizer=tf.keras.optimizers.SGD(), loss='mse')
model.fit(x=fake_data, y=fake_labels, epochs=5)
print(model.get_layer(name='sub_model').a)
如您所見,變量和損失正在更新:
Train on 1000 samples
Epoch 1/5
2020-06-19 19:21:02.475464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
1000/1000 - 1s - loss: 3.9039
Epoch 2/5
1000/1000 - 0s - loss: -3.0905e+00
Epoch 3/5
1000/1000 - 0s - loss: -1.2103e+01
Epoch 4/5
1000/1000 - 0s - loss: -2.6855e+01
Epoch 5/5
1000/1000 - 0s - loss: -5.3408e+01
<tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([-8.13609], dtype=float32)>
Process finished with exit code 0
但是,這仍然是一個非常棘手的方法。 不知道有沒有更優雅穩定的方式來實現同樣的function。
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.