在 TensorFlow2.0.0 中创建自定义激活

Question

I am working on a project on Colab. I want to create a custom activation to use in TensorFlow 2.0.0 as follows:

def custom_activation(x):
  return tf.math.log(x)

model = tf.keras.models.Sequential([
  ... # some layers 
  tf.keras.layers.Dense(10, activation=custom_activation),
  tf.keras.layers.Dense(1)
])

While training I see the following:

Epoch 1/100
      9/Unknown - 6s 674ms/step - loss: nan - mae: nan

Why is the loss and mae nan? From my understanding TF2.0.0 has eager execution enabled. So wouldn't this mean I could evaluate tf.math.log(x) without setting up a Session? The custom activation seems to work for other variations such as tf.math.abs(x). Any idea what I'm doing wrong here? Is it because of Colab or my choice of activation? Any help appreciated. Thanks in advance.

Answer 1

You cannot really just use the logarithm as an activation function, as it is not defined for values x <= 0.0 , so if at any point the Dense layer produces a negative or zero value, the logarithm will produce nan , which then propagates to the loss.

You can easily test this as:

import tensorflow as tf
print(tf.math.log(-1.0))

Which produces:

<tf.Tensor: id=1, shape=(), dtype=float32, numpy=nan>

So its not a programming problem, but a mathematical understanding one.