在 scikit-learn 中填充缺失值后缺少列标签

Question

I was recently tried to fill missing data in a dataset after which the column labels in the dataset disappeared. The dataset is was using looks like:

	Make	Colour	Odometer (KM)	Doors
0	Honda	White	35431.0	4.0
1	BMW	Blue	192714.0	5.0
2	Honda	White	84714.0	4.0
3	Toyota	White	154365.0	4.0
4	Nissan	Blue	181577.0	3.0

The method I used is

from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

cat_imputer=SimpleImputer(strategy='constant',fill_value='missing')
door_imputer=SimpleImputer(strategy='constant',fill_value=4)
num_imputer=SimpleImputer(strategy="mean")

cat_features=["Make","Colour"]
door_features=["Doors"]
num_features=["Odometer (KM)"]

imputer = ColumnTransformer([
    ("cat_imputer",cat_imputer,cat_features),
    ("door_imputer",door_imputer,door_features),
    ("num_imputer",num_imputer,num_features)
])

filled_x=imputer.fit_transform(X)
new_frame=pd.DataFrame(filled_x)

After this, if we view the new_frame it is

	0	1	2	3
0	Honda	White	4.0	35431.04.0
1	BMW	Blue	5.0	192714.0
2	Honda	White	4.0	84714.0
3	Toyota	White	4.0	154365.0
4	Nissan	Blue	3.0	181577.0

So what happened here?

Why are the Doors and Odometer (KM) columns flipped?
Where did the column labels go?
How can I bring them back?

Answer 1

because it generates a whole new filled_x array, which the type is numpy.ndarray , either you assign them back to the original dataframe or to a new dataframe equipt with old dataframe 's columns and index.

df
###
     Make Colour  Odometer (KM)  Doors
0   Honda  White        35431.0    4.0
1     BMW   Blue       192714.0    5.0
2   Honda    NaN        84714.0    4.0
3  Toyota  White       154365.0    4.0
4  Nissan   Blue       181577.0    NaN

I remodify your df with some NaN , so we can check if the imputer works or not.

Imputation

Why are the Doors and Odometer (KM) columns flipped?

Your imputer 's orders differ from your original dataframe 's column order.

cat_imputer=SimpleImputer(strategy='constant',fill_value='missing')
door_imputer=SimpleImputer(strategy='constant',fill_value=4)
num_imputer=SimpleImputer(strategy="mean")

cat_features=["Make","Colour"]
door_features=["Doors"]
num_features=["Odometer (KM)"]

imputer = ColumnTransformer([
    ("cat_imputer",cat_imputer,cat_features),
    ("num_imputer",num_imputer,num_features),     ◀━┓
    ("door_imputer",door_imputer,door_features)   ◀━┛
])

filled_x=imputer.fit_transform(df)

filled_x
###
[['Honda' 'White' 35431.0 4.0]
 ['BMW' 'Blue' 192714.0 5.0]
 ['Honda' 'missing' 84714.0 4.0]
 ['Toyota' 'White' 154365.0 4.0]
 ['Nissan' 'Blue' 181577.0 4.0]]

During the processing, sklearn are dealing with array / ndarray

type(filled_x)
###
numpy.ndarray

you use pd.DataFrame(filled_x) that made the ndarray -format result from imputer.fit_transform() putting into a brand new dataframe without specifying columns and index information.

You may assign them like:

new_frame = pd.DataFrame(filled_x, index=df.index, columns=df.columns)
new_frame
###

And this is a way(though it's not recommended) to assign the result to overwrite the old df

df.loc[:,:] = filled_x

Practically speaking, we might reserve df so you can try different imputation methods.