我無法理解我在 Kaggle 比賽中的訓練和測試數據做錯了什么

Question

在機器學習 kaggle 微課程中，您可以找到這些數據集和代碼來幫助您對比賽進行預測 model： https://www.kaggle.com/ [put your user name here] /exercise-categorical-variables/編輯

它為您提供了兩個數據集：1 個訓練數據集和 1 個測試數據集，您將使用它們來進行預測並提交以查看您在比賽中的排名

所以在：

第 5 步：生成測試預測並提交結果

我寫了這段代碼： EDITED

# Read the data
X = pd.read_csv('../input/train.csv', index_col='Id') 
X_test = pd.read_csv('../input/test.csv', index_col='Id')

# Remove rows with missing target, separate target from predictors
X.dropna(axis=0, subset=['SalePrice'], inplace=True)
y = X.SalePrice
X.drop(['SalePrice'], axis=1, inplace=True)

# To keep things simple, we'll drop columns with missing values
cols_with_missing = [col for col in X.columns if X[col].isnull().any()] 
X.drop(cols_with_missing, axis=1, inplace=True)
X_test.drop(cols_with_missing, axis=1, inplace=True)

#print(X_test.shape, X.shape)

X_test.head()

# Break off validation set from training data
X_train, X_valid, y_train, y_valid = train_test_split(X, y,
                                                      train_size=0.8, test_size=0.2,
                                                      random_state=0)

X_train.head()

#Asses Viability of method ONE-HOT
# Get number of unique entries in each column with categorical data
object_nunique = list(map(lambda col: X_train[col].nunique(), object_cols))
d = dict(zip(object_cols, object_nunique))

# Print number of unique entries by column, in ascending order
sorted(d.items(), key=lambda x: x[1])


# Columns that will be one-hot encoded   ####<<<<I THINK THAT THE PROBLEM STARTS HERE>>>>#####
low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]

##############For X_train

# Apply one-hot encoder to each column with categorical data
OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))
OH_cols_valid = pd.DataFrame(OH_encoder.transform(X_valid[low_cardinality_cols]))

# One-hot encoding removed index; put it back
OH_cols_train.index = X_train.index
OH_cols_valid.index = X_valid.index

# Remove categorical columns (will replace with one-hot encoding)
num_X_train = X_train.drop(object_cols, axis=1)
num_X_valid = X_valid.drop(object_cols, axis=1)

# Add one-hot encoded columns to numerical features
OH_X_train = pd.concat([num_X_train, OH_cols_train], axis=1)
OH_X_valid = pd.concat([num_X_valid, OH_cols_valid], axis=1)

##############For X_test
low_cardinality_cols = [col for col in object_cols if X_test[col].nunique() < 10]

# Apply one-hot encoder to each column with categorical data
OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
#Se não retirar os NAs a linha abaixo dá erro
X_test.dropna(axis = 0, inplace=True)
OH_cols_test = pd.DataFrame(OH_encoder.fit_transform(X_test[low_cardinality_cols]))
#print(OH_cols_test.shape, OH_cols_train.shape)

# One-hot encoding removed index; put it back
OH_cols_test.index = X_test.index


# Remove categorical columns (will replace with one-hot encoding)
num_X_test = X_test.drop(object_cols, axis=1)


# Add one-hot encoded columns to numerical features
OH_X_test = pd.concat([num_X_test, OH_cols_test], axis=1)

#print(OH_X_test.shape ,OH_X_valid.shape)

# Define and fit model
model = RandomForestRegressor(n_estimators=100, criterion='mae', random_state=0)
model.fit(OH_X_train, y_train)

# Get validation predictions and MAE
preds_test = model.predict(OH_X_test)


# Save predictions in format used for competition scoring
output = pd.DataFrame({'Id': X_test.index,
                       'SalePrice': preds_test})
output.to_csv('submission.csv', index=False)

當我嘗試預處理數據集時，我得到不同的行用於訓練數據和測試數據。 然后我無法擬合 de model 並做出預測。

我認為我應該只拆分測試數據集來完成所有這些，但是y比X_test多 1 行，然后我無法進行拆分。

所以我認為我必須使用訓練數據集進行拆分，然后將其擬合為測試數據集的預測

Answer 1

我相信你的問題發生在這一行：

low_cardinality_cols = [col for col in object_cols if X_test[col].nunique() < 10]

您正在為您的獨特列引用 X_test。 按照Kaggle教程，您應該引用 X_train，如下所示：

# Columns that will be one-hot encoded
low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]

您似乎在這一行進一步犯了同樣的錯誤：

OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_test[low_cardinality_cols]))

您已將其標記為 one-hot 編碼訓練列，但您使用的是 X_test 而不是 X_train。 您正在混淆您的訓練和測試集處理，這不是一個好主意。 這一行應該是：

OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))

我建議您再次使用本教程中的代碼塊 go 並確保您的所有數據集和變量都正確匹配，以便您處理正確的訓練和測試數據。