[英]In neural networks, activation is applied by a function or layer?
I am using the Functional API of the TensorFlow/Keras for building a CNN model.我正在使用 TensorFlow/Keras 的功能 API 来构建 CNN model。 In this model, I am trying to apply a custom activation (with constraints) on the output layer.在这个 model 中,我试图在 output 层上应用自定义激活(带约束)。
After going through various resources (1 ,2 ), I am confused about whether the activation needs to be applied by a simple python function or layer.在浏览了各种资源(1 、2 )之后,我很困惑是否需要通过简单的 python function 或层来应用激活。
I tried implementing it by subclassing the Layer class as follows,我尝试通过继承 class 层来实现它,如下所示,
class MapToBounds(layers.Layer):
def __init__(self, lower_bound, upper_bound, **kwargs):
super().__init__(**kwargs)
self.lower_bound = lower_bound
self.upper_bound = upper_bound
def call(self, inputs, *args, **kwargs):
return tf.add(self.lower_bound, tf.multiply(tf.sigmoid(inputs), self.upper_bound))
and called it in the model as并在 model 中将其称为
x = MapToBounds(lower_bound=-3.0, upper_bound=20.0)(x)
where x
is the previous layer instance.其中x
是前一层实例。
My questions are:我的问题是:
training=False
?在这种方法中,我是否必须设置training=False
? Your approach is actually quite clean.你的方法实际上很干净。 You could also just use tf.add
directly on a tensor but a custom layer is more elegant (IMO):您也可以直接在张量上使用tf.add
,但自定义层更优雅(IMO):
import tensorflow as tf
lower_bound=-3.0
upper_bound=20.0
inputs = tf.keras.layers.Input((5, ))
x = tf.keras.layers.Dense(20)(inputs)
outputs = tf.add(lower_bound, tf.multiply(tf.sigmoid(x), upper_bound))
model = tf.keras.Model(inputs, outputs)
model.summary()
Conclusion: Both solutions are valid.结论:两种解决方案都是有效的。 Regarding the flag training=False
, I do not think you have to worry about it unless you want your activation function to act differently during training and inference.关于标志training=False
,我认为您不必担心,除非您希望激活 function 在训练和推理期间表现不同。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.