为测试集估算缺失值

Question

I used adult data here , impute missing values for training data, while I want to apply the same number I get from training data to test data. I must miss something and cannot get it right. My code is as following:

import numpy as np
import pandas as pd
from sklearn.base import TransformerMixin
train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

features = ['age','workclass','fnlwgt','education','educationNum','maritalStatus','occupation','relationship','race','sex','capitalGain','capitalLoss','hoursPerWeek','nativeCountry']

x_train = train[list(features)]
y_train = train['class']
x_test = test[list(features)]
y_test = test['class']

class DataFrameImputer(TransformerMixin):
    def _init_(self):
        """Impute missing values.
        Columns of dtype object are imputed with the most frequent value in column.
        columns of other types are imputed with mean of column"""
    def fit(self, X, y=None):
        self.fill = pd.Series([X[c].value_counts().index[0]
                               if X[c].dtype == np.dtype('O') else X[c].mean() for c in X],
                              index=X.columns)
        return self

    def transform(self, X, y=None):
        return X.fillna(self.fill)


# 2 step transformation, fit and transform
# -------Impute missing values-------------

x_train = pd.DataFrame(x_train)  # x_train is class
x_test = pd.DataFrame(x_test)
x_train_new = DataFrameImputer().fit_transform(x_train)
x_train_new = pd.DataFrame(x_train_new)
# use same value fitted training data to fit test data

for c in x_test:
    if x_test[c].dtype==np.dtype('O'):
        x_test.fillna(x_train[c].value_counts().index[0])
    else:
        x_test.fillna(x_train[c].mean(),inplace=True)

Answer 1

We want to use what we get from training data, apply it to test data, in the previous piece of code, the loop doesn't work, the first column is a column of numbers, so it will fill all the NaNs in the test data as the mean of the first column of training data. Instead, if I use fillna with values, here values is a dictionary, test data will match training data according to categories.

values = {} #declare dict
for c in x_train:
    if x_train[c].dtype==np.dtype('O'):
        values[c]=x_train[c].value_counts().index[0]
    else:
        values[c]=x_train[c].mean()
    values.update({c:values[c]})

x_test_new = x_test.fillna(value=values)