Neural network has <0.001 validation and testing loss but 0% accuracy when doing a prediction

Question

I've been training an MLP to predict the time remaining on an assembly sequence. The Training loss, Validation loss and MSE are all less 0.001, however, when I try to do a prediction with one of the datasets I trained the network with the it can't correctly identify any of the outputs from the set of inputs. What am I doing wrong that is producing this error?

I am also struggling to understand how, when the model is deployed, how do I perform the scaling of the result for one prediction? scaler.inverse_transform won't work because the data for that scaler used during training has been lost as the prediction would be done in a separate script to the training using the model the training produced. Is this information saved in the model builder?

I have tried to change the batch size during training, rounding the time column of the dataset to the nearest second (previously was 0.1 seconds), trained over 50, 100 and 200 epochs and I always end up with no correct predictions. I am also training an LSTM to see which is more accurate but that is also having the same issue. The dataset is split 70-30 training-testing and then training is then split 75-25 into training and validation.

Data scaling and model training code:

def scale_data(training_data, training_data_labels, testing_data, testing_data_labels):
    # Create X and Y scalers between 0 and 1
    x_scaler = MinMaxScaler(feature_range=(0, 1))
    y_scaler = MinMaxScaler(feature_range=(0, 1))

    # Scale training data
    x_scaled_training = x_scaler.fit_transform(training_data)
    y_scaled_training = y_scaler.fit_transform(training_data_labels)

    # Scale testing data
    x_scaled_testing = x_scaler.transform(testing_data)
    y_scaled_testing = y_scaler.transform(testing_data_labels)

    return x_scaled_training, y_scaled_training, x_scaled_testing, y_scaled_testing


def train_model(training_data, training_labels, testing_data, testing_labels, number_of_epochs, number_of_columns):
    model_hidden_neuron_number_list = []
    model_repeat_list = []
    model_error_rate_list = []
    for hidden_layer_1_units in range(int(np.floor(number_of_columns / 2)), int(np.ceil(number_of_columns * 2))):
        print("Training starting, number of hidden units = %d" % hidden_layer_1_units)
        for repeat in range(1, 6):
            print("Repeat %d" % repeat)
            model = k.Sequential()
            model.add(Dense(hidden_layer_1_units, input_dim=number_of_columns,
                        activation='relu', name='hidden_layer_1'))
            model.add(Dense(1, activation='linear', name='output_layer'))
            model.compile(loss='mean_squared_error', optimizer='adam')

            # Train Model
            model.fit(
                training_data,
                training_labels,
                epochs=number_of_epochs,
                shuffle=True,
                verbose=2,
                callbacks=[logger],
                batch_size=1024,
                validation_split=0.25
            )

            # Test Model
            test_error_rate = model.evaluate(testing_data, testing_labels, verbose=0)

            print("Error on testing data is %.3f" % test_error_rate)

            model_hidden_neuron_number_list.append(hidden_layer_1_units)
            model_repeat_list.append(repeat)
            model_error_rate_list.append(test_error_rate)

            # Save Model
            model_builder = tf.saved_model.builder.SavedModelBuilder("MLP/models/{hidden_layer_1_units}/{repeat}".format(hidden_layer_1_units=hidden_layer_1_units, repeat=repeat))

            inputs = {
            'input': tf.saved_model.build_tensor_info(model.input)
            }
            outputs = { 'time_remaining':tf.saved_model.utils.build_tensor_info(model.output)
            }

            signature_def = tf.saved_model.signature_def_utils.build_signature_def(
            inputs=inputs,
            outputs=outputs, method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME
            )

            model_builder.add_meta_graph_and_variables(
                K.get_session(),
                tags=[tf.saved_model.tag_constants.SERVING],
                signature_def_map={tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature_def
                }
            )

        model_builder.save()

And then to do a prediction:

file_name = top_level_file_path + "./MLP/models/19/1/"
    testing_dataset = pd.read_csv(file_path + os.listdir(file_path)[0])
    number_of_rows = len(testing_dataset.index)
    number_of_columns = len(testing_dataset.columns)
    newcol = [number_of_rows]
    max_time = testing_dataset['Time'].max()

    for j in range(0, number_of_rows - 1):
        newcol.append(max_time - testing_dataset.iloc[j].iloc[number_of_columns - 1])

    x_scaler = MinMaxScaler(feature_range=(0, 1))
    y_scaler = MinMaxScaler(feature_range=(0, 1))

    # Scale training data
    data_scaled = x_scaler.fit_transform(testing_dataset)
    labels = pd.read_csv("Labels.csv")
    labels_scaled = y_scaler.fit_transform(labels)

    signature_key = tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY
    input_key = 'input'
    output_key = 'time_remaining'


with tf.Session(graph=tf.Graph()) as sess:
    saved_model = tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], file_name)
        signature = saved_model.signature_def

        x_tensor_name = signature[signature_key].inputs[input_key].name
        y_tensor_name = signature[signature_key].outputs[output_key].name

        x = sess.graph.get_tensor_by_name(x_tensor_name)
        y = sess.graph.get_tensor_by_name(y_tensor_name)

        #np.expand_dims(data_scaled[600], axis=0)
        predictions = sess.run(y, {x: data_scaled})
        predictions = y_scaler.inverse_transform(predictions)
        #print(np.round(predictions, 2))

        correct_result = 0
        for i in range(0, number_of_rows):
            correct_result = 0
            print(np.round(predictions[i]), " ", np.round(newcol[i]))
            if np.round(predictions[i]) == np.round(newcol[i]):
                correct_result += 1
        print((correct_result/number_of_rows)*100)

The output of the first row should 96.0 but it produces 110.0, the last should be 0.1 but is -40.0 when no negatives appear in the dataset.

Answer 1

You can't compute accuracy when you do regression. Compute the mean squared error on the test set as well.

Second, when it comes to the scalers, you always do scaler.fit_transform on the training date so the scaler will compute the parameters (in this case min and max if you use min-max scaler) on the training data. Then, when performing inference on the test set, you should only do scaler.transform prior to feeding the data to the model.