从表格数据训练深度学习 model 时，损失总是 nan

Question

我正在尝试从大约数千个具有 51 个数字特征和标记列的条目的数据集中训练 model，例如：

在训练 model 以预测 3 个标签（候选、误报、确认）时，损失始终为 nan，并且准确度在特定值上稳定得非常快。 编码：

import tensorflow as tf
import numpy as np
import pandas as pd
import sklearn.preprocessing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler, RobustScaler
from sklearn.preprocessing import OrdinalEncoder
from tensorflow import optimizers
from tensorflow.python.keras.layers import Dense, Dropout, Normalization
from tensorflow.python.keras.models import Sequential, Model 


def load_dataset(data_folder_csv):
    # load the dataset as a pandas DataFrame
    data = pd.read_csv(data_folder_csv, header=0)
    # retrieve numpy array
    dataset = data.values

    # split into input (X) and output (y) variables
    X = dataset[:, :-1]
    y = dataset[:, -1]
    print(y)

    # format all fields as floats
    X = X.astype(np.float)
    # reshape the output variable to be one column (e.g. a 2D shape)
    y = y.reshape((len(y), 1))
    return X, y


# prepare input data using min/max scaler.
def prepare_inputs(X_train, X_test):
    oe = RobustScaler().fit_transform(X_train)
    X_train_enc = oe.transform(X_train)
    X_test_enc = oe.transform(X_test)
    return X_train_enc, X_test_enc


# prepare target
def prepare_targets(y_train, y_test):
    le = LabelEncoder()
    ohe = OneHotEncoder()
    le.fit(y_train)
    le.fit(y_test)
    y_train_enc = ohe.fit_transform(y_train).toarray()
    y_test_enc = ohe.fit_transform(y_test).toarray()
    return y_train_enc, y_test_enc



X, y = load_dataset("csv_ready.csv")

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)

print('Train', X_train.shape, y_train.shape)
print('Test', X_test.shape, y_test.shape)

X_train_enc, X_test_enc = X_train, X_test
print('Finished preparing inputs.'

# prepare output data
y_train_enc, y_test_enc = prepare_targets(y_train, y_test)

norm_layer = Normalization()
norm_layer.adapt(X)
model = Sequential()
model.add(Dense(128, input_dim=X_train.shape[1], activation="tanh", kernel_initializer='he_normal'))
model.add(Dropout(0.2))
model.add(Dense(64, input_dim=X_train.shape[1], activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(32, input_dim=X_train.shape[1], activation='relu'))
model.add(Dense(3, activation='sigmoid'))

opt = optimizers.Adam(lr=0.01, decay=1e-6)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
model.summary()

model.fit(X_train, y_train_enc, epochs=20, batch_size=128, verbose=1, use_multiprocessing=True)

_, accuracy = model.evaluate(X_test, y_test_enc, verbose=0)
print('Accuracy: %.2f' % (accuracy * 100))

我尝试增加/减少学习率，更改优化器，降低和增加神经元和层的数量，以及使用批量大小，但似乎没有什么能让 model 获得好的结果。 我想我在这里遗漏了一些东西，但不能指望它。 结果示例：

编辑：来自 csv 的更多行：

EDIT2：也尝试了l2正则化并且没有做任何事情。

Answer 1

原因之一：检查您的数据集是否具有NaN值。 NaN值可能会在学习时导致 model 出现问题。

您的代码中的一些主要错误：

• 您正在使用sigmoid激活 function 而不是softmax用于具有 3 个神经元的 output 层
• 您在使用错误的编码器时同时拟合训练集和测试集。 您应该为您的训练数据fit_transform并且只对测试集使用transform
• 此外，您对所有错误的层都使用了输入，只有第一层应该接受输入张量。
• 您忘记对X_train和X_test使用prepare_inputs function
• 您的 model 应该适合X_train_enc而不是X_train

改用这个

import tensorflow as tf
import numpy as np
import pandas as pd
import sklearn.preprocessing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler, MinMaxScaler
from sklearn.preprocessing import OrdinalEncoder
from tensorflow import optimizers
from tensorflow.python.keras.layers import Dense, Dropout, Normalization
from tensorflow.python.keras.models import Sequential, Model 


def load_dataset(data_folder_csv):
    # load the dataset as a pandas DataFrame
    data = pd.read_csv(data_folder_csv, header=0)
    # retrieve numpy array
    dataset = data.values

    # split into input (X) and output (y) variables
    X = dataset[:, :-1]
    y = dataset[:, -1]
    print(y)

    # format all fields as floats
    X = X.astype(np.float)
    # reshape the output variable to be one column (e.g. a 2D shape)
    y = y.reshape((len(y), 1))
    return X, y

# prepare input data using min/max scaler.
def prepare_inputs(X_train, X_test):
    oe = MinMaxScaler()
    X_train_enc = oe.fit_transform(X_train)
    X_test_enc = oe.transform(X_test)
    return X_train_enc, X_test_enc

# prepare target
def prepare_targets(y_train, y_test):
    le = LabelEncoder()
    ohe = OneHotEncoder()
    y_train = le.fit_transform(y_train)
    y_test = le.transform(y_test)
    y_train_enc = ohe.fit_transform(y_train).toarray()
    y_test_enc = ohe.transform(y_test).toarray()
    return y_train_enc, y_test_enc



X, y = load_dataset("csv_ready.csv")

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=1)

print('Train', X_train.shape, y_train.shape)
print('Test', X_test.shape, y_test.shape)

#prepare_input function missing here
X_train_enc, X_test_enc = prepare_inputs(X_train, X_test)
print('Finished preparing inputs.')

# prepare output data
y_train_enc, y_test_enc = prepare_targets(y_train, y_test)

model = Sequential()
model.add(Dense(128, input_dim=X_train.shape[1], activation="relu")) 
model.add(Dropout(0.2))
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(128, activation='relu'))
model.add(Dense(3, activation='softmax'))

#opt = optimizers.Adam(lr=0.01, decay=1e-6)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

model.fit(X_train_enc, y_train_enc, epochs=20, batch_size=32, verbose=1, use_multiprocessing=True)

_, accuracy = model.evaluate(X_test_enc, y_test_enc, verbose=0)
print('Accuracy: %.2f' % (accuracy * 100))

Answer 2

您想将 model 定义更改为：

model = Sequential()
model.add(Dense(128, input_shape=X_train.shape[1:], activation="tanh", kernel_initializer='he_normal'))
model.add(Dropout(0.2))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='relu'))
model.add(Dense(3, activation='softmax'))

您只需定义第一层的输入形状，Keras 将自动确定后续层的正确形状。 在定义第一个维度 input_shape 时，您忽略了批量大小，因此input_shape=X_train.shape[1:] 。

sigmoid激活实际上会起作用（因为 output 将在 0 和 1 之间变化），但您真正想要的是softmax激活（确保所有输出总和为 1，这是概率所决定的 - 发生某事的概率是 100%，而不是sigmoid最终可能给你的 120%）。

此外，您没有在任何地方使用您的LabelEncoder 。 我想你的意思是这样的：

def prepare_targets(y_train, y_test):
    le = LabelEncoder()
    ohe = OneHotEncoder()

    # teach the label encoder our labels
    le.fit(y_train)

    # turn our strings into integers
    y_train_transformed = le.transform(y_train)
    y_test_transformed = le.transform(y_test)

    # turn our integers into one-hot-encoded arrays
    y_train_enc = ohe.fit_transform(y_train_transformed).toarray()
    y_test_enc = ohe.transform(y_test_transformed).toarray()
    return y_train_enc, y_test_enc