Keras DNN prediction model Accuracy is not improving

Question

I am trying to train the Keras DNN model for prediction using LUT Data. I have normalized the data and split into training, testing, and validation part. I encountered a problem with my tanning and validation accuracy that stays (almost) the same. The accuracy is always stuck at (0.1431).

I've tried many different hyperparameters, including changing the activation functions to tanh and relu, and I've tried adding a batch normalization layer after the first dense layer, I've used SGD optimizer (changed the learning rate, momentum, even tried changing the optimizer to Adam), tried different loss functions, added/removed dropout layers.

import pandas as pd
import numpy as np
import tensorflow as tf
from sklearn import preprocessing
from sklearn.metrics import explained_variance_score, \
     mean_absolute_error, \
     median_absolute_error
from sklearn.model_selection import train_test_split
##########################################################
# for DNN model
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from tensorflow import feature_column
import os
import datetime
from sklearn.preprocessing import StandardScaler,MinMaxScaler

df=pd.read_csv("..../LUT.csv")

Normalized the data (between 0 to 1)

scaler = MinMaxScaler()
df[df.columns] = scaler.fit_transform(df[df.columns].to_numpy())
# X will be a pandas dataframe of all columns except meantempm
X = df[[col for col in df.columns if col != 'TT']]
# y will be a pandas series of the meantempm
Y = df['TT']

split data into training set and a temporary set using sklearn.model_selection.traing_test_split

X_train, X_tmp, y_train, y_tmp = train_test_split(X, Y,  test_size=0.20, random_state=23)
# take the remaining 20% of data in X_tmp, y_tmp and split them evenly
X_test, X_val, y_test, y_val = train_test_split(X_tmp, y_tmp, test_size=0.5, random_state=23)

X_train.shape, X_test.shape, X_val.shape
print("Training instances   {}, Training features   {}".format(X_train.shape[0], X_train.shape[1]))
print("Validation instances {}, Validation features {}".format(X_val.shape[0], X_val.shape[1]))
print("Testing instances    {}, Testing features    {}".format(X_test.shape[0], X_test.shape[1]))

Create Keras dense features layer out of array with TensorFlow encodings. We will use this layer during Keras model construction to define model training features:

feature_columns = [feature_column.numeric_column(x) for x in X.columns]
feature_layer = tf.keras.layers.DenseFeatures(feature_columns)
feature_layer

Function for create datasets for tensorflow format

def df_to_dataset(x,y, shuffle=True, batch_size=32):
    dataframe = x.copy()
    labels = y.copy()
    ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
    if shuffle:
      ds = ds.shuffle(buffer_size=len(dataframe))
    ds = ds.batch(batch_size)
   return ds

Next, convert Pandas dataframe to tf.data with the help of the utility function:

  batch_size = 250
  train_ds = df_to_dataset(X_train,y_train, batch_size=batch_size)
  val_ds = df_to_dataset(X_val,y_val, shuffle=False, batch_size=batch_size)
  test_ds = df_to_dataset(X_test,y_test, shuffle=False, batch_size=batch_size)

Model:

  #relu,sigmoid,tanh
  def get_compiled_model():
     model = keras.Sequential([
         feature_layer,
         layers.Dense(50, activation="tanh"),
         tf.keras.layers.Dropout(0.1),
         layers.Dense(35, activation='tanh'),
         layers.Dense(20, activation='tanh'),
         # layers.Dense(100, activation='tanh'),
         # tf.keras.layers.Dropout(0.1),
         layers.Dense(1,activation="linear")
            ])

  # Compile the model with the specified loss function.
   model.compile(optimizer=keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08),
            loss='mse',
            metrics=["accuracy",'mape',"RootMeanSquaredError"])
   return model

Train the model:

  # Callbacks time
  logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
  tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)
   es = EarlyStopping(monitor='val_loss', patience=10)
   mcp = ModelCheckpoint(filepath='best_model_GPU_V1.h5', monitor='val_loss', save_best_only=True)

  # Create a MirroredStrategy.
   strategy = tf.distribute.MirroredStrategy()
   print("Number of devices: {}".format(strategy.num_replicas_in_sync))

  # Open a strategy scope.
  with strategy.scope():
   # Everything that creates variables should be under the strategy scope.
   # In general this is only model construction & `compile()`.
    model = get_compiled_model()

  # Train the model on all available devices.
   EPOCHS = 50
   history = model.fit(train_ds,
          epochs=EPOCHS,
          # steps_per_epoch=1000,
          callbacks=[tensorboard_callback,es,mcp],
          validation_data=val_ds
          )

Training Result:

 Epoch 40/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0202 - loss: 
 4.0961e-04 - mape: 1093214.5000 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0124 - val_loss: 
 1.5268e-04 - val_mape: 509855.8438 - val_accuracy: 0.1464
 Epoch 41/50
 621/621 [==============================] - 4s 6ms/step - root_mean_squared_error: 0.0201 - loss: 
 4.0516e-04 - mape: 1089531.5000 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0115 - val_loss: 
 1.3204e-04 - val_mape: 527368.5000 - val_accuracy: 0.1464
 Epoch 42/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0199 - loss: 
 3.9764e-04 - mape: 1048669.6250 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0107 - val_loss: 
 1.1494e-04 - val_mape: 543746.5625 - val_accuracy: 0.1464
 Epoch 43/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0198 - loss: 
 3.9081e-04 - mape: 1053232.5000 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0111 - val_loss: 
 1.2281e-04 - val_mape: 659315.5000 - val_accuracy: 0.1464
 Epoch 44/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0196 - loss: 
 3.8481e-04 - mape: 1046033.1250 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0132 - val_loss: 
 1.7504e-04 - val_mape: 944899.8125 - val_accuracy: 0.1464
 Epoch 45/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0196 - loss: 
 3.8521e-04 - mape: 1033596.6875 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0113 - val_loss: 
 1.2671e-04 - val_mape: 535661.8750 - val_accuracy: 0.1464
 Epoch 46/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0196 - loss: 
 3.8274e-04 - mape: 1045924.3125 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0101 - val_loss: 
 1.0106e-04 - val_mape: 587111.2500 - val_accuracy: 0.1464
 Epoch 47/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0195 - loss: 
 3.7925e-04 - mape: 1038761.8125 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0112 - val_loss: 
 1.2610e-04 - val_mape: 474619.3125 - val_accuracy: 0.1464
 Epoch 48/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0194 - loss: 
 3.7453e-04 - mape: 1024884.4375 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0106 - val_loss: 
 1.1254e-04 - val_mape: 537549.6250 - val_accuracy: 0.1464
 Epoch 49/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0193 - loss: 
 3.7414e-04 - mape: 1033414.7500 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0122 - val_loss: 
 1.4766e-04 - val_mape: 475745.0000 - val_accuracy: 0.1464
 Epoch 50/50
 621/621 [==============================] - 4s 7ms/step - root_mean_squared_error: 0.0194 - loss: 
 3.7510e-04 - mape: 1027084.1250 - accuracy: 0.1431 - val_root_mean_squared_error: 0.0094 - val_loss: 
 8.9167e-05 - val_mape: 506829.9062 - val_accuracy: 0.1464

Tranning graph

I am very confused about how to improve the accuracy of the DNN prediction model. If anyone gives me suggestions I will be very thankful.

Answer 1

Your loss is MSE , which is like the problem you are solving is the Regression. Accuracy is a metric for classification, and that is the reason you are getting an accuracy which is barely changing. I suggest evaluating your model with MSE as well if you want to go with Regression

In case you do really want to go with the classification,(which I think is not a good idea regarding your continuous target values) you need to change your loss from MSE to cross-entropy , whether binary or categorical depending if you are solving a binary or a multi-class classification task