[英]How can I assign a custom alpha for Keras relu function, for each node in a layer?
I would like to add a node-specific variable to each Keras activation function.我想为每个 Keras 激活函数添加一个特定于节点的变量。 I would like each node to compute the activation value (output) with a different value (
alpha
).我希望每个节点都使用不同的值 (
alpha
) 计算激活值(输出)。
This can be done globally , for example with the alpha
parameter for the relu activation function ( link ):这可以全局完成,例如使用 relu 激活函数的
alpha
参数( 链接):
# Build Model
...
model.add(Dense(units=128))
model.add(Activation(lambda x: custom_activation(x, alpha=0.1)))
...
I can also write a custom activation function, but the alpha
parameter is also global .我也可以写一个自定义的激活函数,但是
alpha
参数也是全局的。 ( link ) ( 链接)
# Custom activation function
def custom_activation(x, alpha=0.0):
return (K.sigmoid(x + alpha))
# Build Model
...
model.add(Dense(units=128))
model.add(Activation(lambda x: custom_activation(x, alpha=0.1)))
...
Inside the custom function I only currently have access to the following variables:在自定义函数中,我目前只能访问以下变量:
(Pdb) locals()
{'x': <tf.Tensor 'dense/Identity:0' shape=(None, 128) dtype=float32>, 'alpha': 0.1}
I would like to use a custom activation function, but for alpha
to be unique to each node in the network.我想使用自定义激活函数,但
alpha
对于网络中的每个节点都是唯一的。 For example, if there are 128 units in the layer, then I would like there also to be 128 values of alpha, one for each unit / node.例如,如果层中有 128 个单元,那么我希望也有 128 个 alpha 值,每个单元/节点一个。 I would then like the activation function to
然后我希望激活函数
How do I create an alpha
value that is unique to each unit / node in a layer?如何创建图层中每个单元/节点唯一的
alpha
值?
I would not recommend to use lambda layer for that one, it is to hackish.我不建议为那个使用 lambda 层,它太骇人听闻了。 I suggest you write your own layer as follows:
我建议你写你自己的图层如下:
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
# Custom layer
class CustomAct(tf.keras.layers.Layer):
def __init__(self):
super(CustomAct, self).__init__()
def build(self, input_shape):
self.alpha = self.add_weight(name='alpha',
shape=[input_shape[1], ],
initializer='uniform',
trainable=True)
super(CustomAct, self).build(input_shape)
def call(self, x):
return tf.sigmoid(x+self.alpha)
def get_alpha(self):
return self.alpha
inputs = np.random.random([16, 32]).astype(np.float32)
# Model
model = tf.keras.models.Sequential()
model.add(tf.keras.Input(inputs.shape[-1]))
model.add(tf.keras.layers.Dense(128))
model.add(CustomAct())
# Test
model.compile(loss="MSE")
alpha_after_initialization = model.layers[-1].get_alpha()
plt.plot(alpha_after_initialization.numpy())
x = np.random.random([18, 32])
y = np.random.random([18, 128])
for _ in range(20):
model.fit(x, y)
out_after_20_steps = alpha_after_initialization = model.layers[-1].get_alpha()
plt.plot(alpha_after_initialization.numpy())
plt.show()
Of course, you should change all tf refernces to your keras ones.当然,您应该将所有 tf 引用更改为您的 keras 引用。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.