如何使用 OneHotEncoder 和 LabelEncoder 预处理看不见的数据以匹配训练集？

Question

I wrote a classifier and did a data preprocessing (it was categorical data) with scikit learn with LabelEncoder (LE) and OneHotEncoder (OHE) and it work great on train and test data. Now, i want to make predictions on new data. My question: how I convert the new data with LE and OHE on the same style (in a lack of a better word) as the training data? My code so far:

labelencoder_X = LabelEncoder()

X[:, 0] = labelencoder_X.fit_transform(X[:, 0])
X[:, 1] = labelencoder_X.fit_transform(X[:, 1])
onehotencoder = OneHotEncoder(categorical_features='all')# se quiser em uma coluna coloca categorical_features=[0],
#onde [0] é o index da coluna e se quiser em todas as colunas coloca 'all
X = onehotencoder.fit_transform(X).toarray()

labelencoder_y = LabelEncoder()
y = labelencoder_y.fit_transform(y)

what I tried without sucess:

new_pred = np.array(['car','male'])
labelencoder_new_pred = LabelEncoder()
new_pred = labelencoder_new_pred.fit_transform(new_pred) #also tried new_pred = labelencoder_X.fit_transform(new_pred) 
onehotencoder2 = OneHotEncoder(categorical_features='all',n_values=29)

new_pred = onehotencoder2.fit_transform(new_pred).toarray()#also tried new_pred = onehotencoder.fit_transform(new_pred).toarray()

z = cfl.predict(new_pred)

The results of this:

1. The result is always the same, even changing the new_pred data with an equal data found in the train set
2. It produced OHE different that on the train set

What I'm missing here? Thks!

Answer 1

You'll have to store (ie pickle) your fitted LabelEncoders and OneHotEncoder. Check it out here: model persistence

When you receive new data, you'll transform them via the already-fitted LabelEncoders and OneHotEncoder and then use your trained model to make the predictions. This way, the produced data will be in the exact format your models expects them to be,

Answer 2

You were trying to use the same instance to categorize two different categories. Try using something like below

labelencoder_X_1 = LabelEncoder()
labelencoder_X_2 = LabelEncoder()

X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])

Now you can use

new_data[:,1] = labelencoder_X_2.fit_transform(new_data[:,1])

where new_data is the sample data that you want to preprocess for prediction.

Similarly you can use the same method for Encoding