低訓練損失，高驗證損失和低驗證准確率

Question

我正在嘗試使用categorical_crossentropy對多類分類問題（心臟病數據集）使用 Keras（TensorFlow 作為后端）獲得良好的准確性。 我的模型可以達到很好的訓練精度，但驗證精度低（驗證損失高）。 我嘗試過過擬合的解決方案（例如，歸一化、dropout、正則化等），但我仍然遇到同樣的問題。 到目前為止，我一直在嘗試優化器、損失、時代和批量大小，但沒有成功。 這是我正在使用的代碼：

import pandas as pd
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.optimizers import SGD,Adam
from keras.layers import Dense, Dropout
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from keras.utils import to_categorical
from sklearn.model_selection import train_test_split
from keras.models import load_model
from keras.regularizers import l1,l2
# fix random seed for reproducibility
np.random.seed(5)
data = pd.read_csv('ProcessedClevelandData.csv',delimiter=',',header=None)
#Missing Values
Imp=SimpleImputer(missing_values=np.nan,strategy='mean',copy=True)
Imp=Imp.fit(data.values)
Imp.transform(data)
X = data.iloc[:, :-1].values
y=data.iloc[:,-1].values

y=to_categorical(y)
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.1)
scaler = StandardScaler()
X_train_norm = scaler.fit_transform(X_train)
X_test_norm=scaler.transform(X_test)
# create model
model = Sequential()
model.add(Dense(13, input_dim=13, activation='relu',use_bias=True,kernel_regularizer=l2(0.0001)))
#model.add(Dropout(0.05))
model.add(Dense(9, activation='relu',use_bias=True,kernel_regularizer=l2(0.0001)))
#model.add(Dropout(0.05))
model.add(Dense(5,activation='softmax'))
sgd = SGD(lr=0.01, decay=0.01/32, nesterov=False)
# Compile model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])#adam,adadelta,
print(model.summary())
history=model.fit(X_train_norm, y_train,validation_data=(X_test_norm,y_test), epochs=1200, batch_size=32,shuffle=True)
# list all data in history
print(history.history.keys())
# summarize history for accuracy
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

這是輸出的一部分，您可以在其中看到上述行為：

Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 13)                182       
_________________________________________________________________
dense_2 (Dense)              (None, 9)                 126       
_________________________________________________________________
dense_3 (Dense)              (None, 5)                 50        
=================================================================
Total params: 358
Trainable params: 358
Non-trainable params: 0
_________________________________________________________________

Train on 272 samples, validate on 31 samples
Epoch 1/1200

 32/272 [==>...........................] - ETA: 21s - loss: 1.9390 - acc: 0.1562
272/272 [==============================] - 3s 11ms/step - loss: 2.0505 - acc: 0.1434 - val_loss: 2.0875 - val_acc: 0.1613
Epoch 2/1200

 32/272 [==>...........................] - ETA: 0s - loss: 1.6747 - acc: 0.2188
272/272 [==============================] - 0s 33us/step - loss: 1.9416 - acc: 0.1544 - val_loss: 1.9749 - val_acc: 0.1290
Epoch 3/1200

 32/272 [==>...........................] - ETA: 0s - loss: 1.7708 - acc: 0.2812
272/272 [==============================] - 0s 37us/step - loss: 1.8493 - acc: 0.1801 - val_loss: 1.8823 - val_acc: 0.1290
Epoch 4/1200

 32/272 [==>...........................] - ETA: 0s - loss: 1.9051 - acc: 0.2188
272/272 [==============================] - 0s 33us/step - loss: 1.7763 - acc: 0.1949 - val_loss: 1.8002 - val_acc: 0.1613
Epoch 5/1200

 32/272 [==>...........................] - ETA: 0s - loss: 1.6337 - acc: 0.2812
272/272 [==============================] - 0s 33us/step - loss: 1.7099 - acc: 0.2426 - val_loss: 1.7284 - val_acc: 0.1935
Epoch 6/1200
....
 32/272 [==>...........................] - ETA: 0s - loss: 0.0494 - acc: 1.0000
272/272 [==============================] - 0s 37us/step - loss: 0.0532 - acc: 1.0000 - val_loss: 4.1031 - val_acc: 0.5806
Epoch 1197/1200

 32/272 [==>...........................] - ETA: 0s - loss: 0.0462 - acc: 1.0000
272/272 [==============================] - 0s 33us/step - loss: 0.0529 - acc: 1.0000 - val_loss: 4.1174 - val_acc: 0.5806
Epoch 1198/1200

 32/272 [==>...........................] - ETA: 0s - loss: 0.0648 - acc: 1.0000
272/272 [==============================] - 0s 37us/step - loss: 0.0533 - acc: 1.0000 - val_loss: 4.1247 - val_acc: 0.5806
Epoch 1199/1200

 32/272 [==>...........................] - ETA: 0s - loss: 0.0610 - acc: 1.0000
272/272 [==============================] - 0s 29us/step - loss: 0.0532 - acc: 1.0000 - val_loss: 4.1113 - val_acc: 0.5484
Epoch 1200/1200

 32/272 [==>...........................] - ETA: 0s - loss: 0.0511 - acc: 1.0000
272/272 [==============================] - 0s 29us/step - loss: 0.0529 - acc: 1.0000 - val_loss: 4.1209 - val_acc: 0.5484

Answer 1

一個問題可能是您的數據在訓練和測試拆分中分布不均（如評論中所述）。 嘗試查看分布是否不均勻，如果是，請嘗試不同的種子。 我之前在一個小型醫療數據集上遇到過類似的問題。 數據集越小，分割數據集不能准確表示真實分布的可能性就越大。

編輯：根據您設置種子的方式，您可以使用np.random.seed(my_seed)為 numpy 設置它，或使用random.seed(my_seed)為 python 模塊設置它，或為 keras 設置它，請遵循他們的文檔.

Answer 2

通過將驗證大小增加到大約 30% 來幫助自己，除非您確實擁有大量數據集。 甚至經常使用 50/50。

請記住，良好的 loss 和 acc 與不良的 val_loss 和 val_acc 意味着過度擬合。

試試這個基本解決方案：

from keras.callbacks import EarlyStopping, ReduceLROnPlateau

early_stop = EarlyStopping(monitor='val_loss',patience=10)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.1,
                              patience=6, verbose=1, mode='auto',
                              min_delta=0.0001, cooldown=0, min_lr=1e-8)

history = model.fit(X,y,num_epochs=666,callbacks=[early_stop,reduce_lr])

希望有幫助！