Custom tf.keras metrics unexpected result

Question

I'm doing a regression task and I need to scale my target, however during the evaluation I want to measure my net's performance on the original target, thus I made these custom metrics:

def RMSE_rescaled(max_output, min_output):
    def RMSE(Y_true, Y_pred):
        Y_true = tf.add(tf.math.multiply(Y_true,(max_output - min_output)), min_output)
        Y_pred = tf.add(tf.math.multiply(Y_pred,(max_output - min_output)), min_output)
        return tf.math.sqrt(keras.metrics.mean_squared_error(Y_true, Y_pred))
    return RMSE

def MAE_rescaled(max_output, min_output):
    def MAE(Y_true, Y_pred):
        Y_true = tf.add(tf.math.multiply(Y_true,(max_output - min_output)), min_output)
        Y_pred = tf.add(tf.math.multiply(Y_pred,(max_output - min_output)), min_output)
        return keras.metrics.mean_absolute_error(Y_true, Y_pred)
    return MAE

checkpoint_cb = keras.callbacks.ModelCheckpoint("resnet_lstm{epoch:02d}.h5", period=1)

tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)


model.compile(loss=loss, optimizer=optimizer, 
              metrics=[RMSE_rescaled(max_output=max_output,min_output=min_output), 
                       MAE_rescaled(max_output=max_output,min_output=min_output)])

history = model.fit(data[:,0:divisore], data[:,divisore:], epochs=5,
                    validation_data=(val[:,0:divisore], val[:,divisore:]),
                    callbacks=[checkpoint_cb, tensorboard_cb])

The problem is I get always the same RMSE and MAE, the strange thing is that the problem is tf.math.sqrt(). Function: RMSE_rescaled.

Indeed, if I remove it I get different values.

def RMSE_rescaled(max_output, min_output):
    def RMSE(Y_true, Y_pred):
        Y_true = tf.add(tf.math.multiply(Y_true,(max_output - min_output)), min_output)
        Y_pred = tf.add(tf.math.multiply(Y_pred,(max_output - min_output)), min_output)
        return keras.metrics.mean_squared_error(Y_true, Y_pred))
    return RMSE

As You can see in the first image, I get RMSE 22.1573 which is equal to MAE and it's clearly wrong.

While in the second image I get MAE 19.6551 and RMSE 790.0024, this is correct, indeed if I try to calculate sqrt(RMSE) I get 28.10 which is different from MAE.

In case you need it: data shape is [270000,250,1].

Could you help me understand what am I doing wrong? Thank you.

Ps: As u can see I early stopped the training, however, if I utilize tf.math.sqrt() i always get MAE = RMSE despite the training step. Furthermore, could you explain why do i get that warning? It's not a real problem, it's just i would like to know what's the problem with my callbacks, even tho i guess it's because i'm saving too often

Answer 1

In my dummy example seems to work (I'm using TF 2.2)

max_output = 3.
min_output = 1.

def RMSE(Y_true, Y_pred):
    Y_true = Y_true*(max_output - min_output) + min_output
    Y_pred = Y_pred*(max_output - min_output) + min_output
    return tf.math.sqrt(tf.reduce_mean(tf.square(Y_true - Y_pred)))

def MAE(Y_true, Y_pred):
    Y_true = Y_true*(max_output - min_output) + min_output
    Y_pred = Y_pred*(max_output - min_output) + min_output
    return tf.reduce_mean(tf.abs(Y_true - Y_pred))


inp = Input((100))
x = Dense(50)(inp)
x = Dense(1)(x)
m = Model(inp, x)
m.compile('adam','mse', metrics=[RMSE,MAE])

m.fit(np.random.uniform(0,10, (10,100)), 
      np.random.uniform(0,10, 10), epochs=10, verbose=2)

OUTPUT:

Epoch 1/10
1/1 - 0s - loss: 90.4590 - RMSE: 19.0220 - MAE: 15.6778
Epoch 2/10
1/1 - 0s - loss: 37.0396 - RMSE: 12.1720 - MAE: 8.5783
Epoch 3/10
1/1 - 0s - loss: 26.2672 - RMSE: 10.2503 - MAE: 8.8775
Epoch 4/10
1/1 - 0s - loss: 38.6563 - RMSE: 12.4348 - MAE: 10.4935