Keras CuDNNLSTM 隐式激活函数？

Question

While intf.keras.layers.LSTM there is an activation parameter (default tanh )
the CuDNNLSTM doesn't have any, while having a activity_regularizer parameter.
Am I missing something?

CuDNNLSTM is not a tf.keras.layers.LSTM wrapper, but a standalone entity, so how do I set the activation function for it?

Answer 1

CuDNNLSTM has a hardcoded tanh activation. As far as I am aware, there is no way for this to be changed. If you need some other activation, you're stuck using a regular LSTM layer.

Alternately, if you only need a specific output activation for your model, eg softmax, you can stick on a Dense layer and then put the activation after that. Here's a snippet of how I'm doing that for my particular case:

x = CuDNNLSTM(256, return_sequences=True)(x)
x = TimeDistributed(Dense(8))(x)
x = Softmax(axis=2)(x)

Further reading:

• https://github.com/keras-team/keras/issues/8510
• https://devblogs.nvidia.com/optimizing-recurrent-neural-networks-cudnn-5/

Answer 2

You can reverse the tanh activation, then apply whatever activation u want!

for example:

x = LSTM(200,kernel_initializer=VarianceScaling(2))(x)

o = 1-1e-12 #float a little bit smaller than 1 in case LSTM returns -1 or 1
#x = Lambda(lambda x: 0.5*tf.math.log((1+x*o)/(1-x*o)))(x) #Tanh-¹
x = Lambda(lambda x: tf.math.atanh(x*o))(x)

x = LeakyReLU(alpha=0.01)(x) #apply new activation