Why does the result from evaluate() differ from last epoch result?

Question

I have a very simple neural-network which works in 250 epochs and in the last epoch it shows the mae = 0.1397 , however, if i try to get the model.evaluate((m * test_x + b), predict_y)) then the mae is about 44009.296875 .

Why there is such big difference ?

Here is my code:

import tensorflow as tf
from tensorflow.keras import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import plot_model

import numpy as np
import matplotlib.pyplot as plt

train_x = np.arange(2000)
m = 5
b = 4

train_y = m * train_x + b

# -----------------------------------------------------
# Create a Sequential Nerual Network
model = tf.keras.Sequential()
model.add(Input(shape=(1,), name="input_layer"))
model.add(Dense(10, activation="relu"))
model.add(Dense(1, activation=None, name="output_layer"))

# -----------------------------------------------------
# Compile the model
model.compile(loss=tf.keras.losses.mae,
              optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),
              metrics=["mae"])

# -----------------------------------------------------
# Train the model
model.fit(train_x, train_y, epochs=250)

# -----------------------------------------------------
# Test the model

test_x = np.arange(2000, 2400)
predict_y = model.predict([test_x])
# ------------------------------------------------------
# Evaluation
print("Evaluate Testing : ", model.evaluate((m * test_x + b), predict_y))

Answer 1

I am not too sure if you are using the model.evaluate method correctly. Similar to the model.fit method, when evaluating your model you should provide x and y values. I get quite similar results when running this code snippet:

import tensorflow as tf
from tensorflow.keras import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import plot_model

import numpy as np
import matplotlib.pyplot as plt

train_x = np.arange(2000)
m = 5
b = 4
train_y = m * train_x + b

# -----------------------------------------------------
# Create a Sequential Nerual Network
model = tf.keras.Sequential()
model.add(Input(shape=(1,), name="input_layer"))
model.add(Dense(10, activation="relu"))
model.add(Dense(1, activation=None, name="output_layer"))

# -----------------------------------------------------
# Compile the model
model.compile(loss=tf.keras.losses.mae,
              optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),
              metrics=["mae"])

# -----------------------------------------------------
# Train the model
model.fit(train_x, train_y, epochs=5, batch_size=32)

# -----------------------------------------------------
# Test the model
test_x = np.arange(2000)
predict_y = model.predict([test_x])
# ------------------------------------------------------
# Evaluation
print("Evaluate Testing : ", model.evaluate(test_x,  m * test_x + b, batch_size=32))

Epoch 1/5
63/63 [==============================] - 1s 3ms/step - loss: 4978.4922 - mae: 4978.4922
Epoch 2/5
63/63 [==============================] - 0s 3ms/step - loss: 4954.3252 - mae: 4954.3252
Epoch 3/5
63/63 [==============================] - 0s 3ms/step - loss: 4929.9980 - mae: 4929.9980
Epoch 4/5
63/63 [==============================] - 0s 3ms/step - loss: 4905.5146 - mae: 4905.5146
Epoch 5/5
63/63 [==============================] - 0s 3ms/step - loss: 4880.8120 - mae: 4880.8120
63/63 [==============================] - 0s 2ms/step - loss: 4868.2192 - mae: 4868.2192
Evaluate Testing :  [4868.21923828125, 4868.21923828125]

Due to the stochastic nature of the whole process, the results will naturally vary a little.

Answer 2

You can evaluate results manually:

np.sqrt(np.mean((m * test_x + b) - predict_y)) # manually computing mae

>>> 0.35962627085281185

It doesn't make any sense why is that such a big difference here. At least we can assume, that your predictions are not so bad.

There is similar issue in github: https://github.com/keras-team/keras/issues/6977 . And kind of official answer here: https://keras.io/getting_started/faq/#why-is-my-training-loss-much-higher-than-my-testing-loss

Why is my training loss much higher than my testing loss?

A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time. They are reflected in the training time loss but not in the test time loss.

Besides, the training loss that Keras displays is the average of the losses for each batch of training data, over the current epoch. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. This can bring the epoch-wise average down. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss.