How to handle "unseen" categorical variables with one hot encoding in sklearn

Question

I have a training data (df_train) in which I applied 3rd polynomial to variable x1 and also one hot encoding approach to color variables. The goal is to get the coefficient for each independent variable and predict the Y (target variable) in the testing data (df_test).

As you can see from the below code, training data only has 3 colors (green, red and purple) where testing data has 2 additional colors which are yellow and black. In such case, yellow and black are unseen categorical variables in the testing data.

I have done some researches and found good amount of tutorials/posts on handling unseen categorical variables but I couldn't find any specific examples similar to my case with sklearn Pipeline, ColumnTransformer and PolynomialFeatures.

Any suggestion and advice specific to my use case are greatly appreciated.

import pandas as pd
import numpy as np

from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

# Training data
x1 = [28.0, 29.0, 12.0, 12.0, 42.0]
x2 = [0.36, 0.53, 0.45, 0.48, 0.4] 
y = [59.5833333333333, 59.5833333333333, 10.0, 10.0, 47.0833333333333] 
color = ['green','red','red','purple','purple']

df_train = pd.DataFrame({
'x1': x1,
'x2' :x2,
'y': y,
'color':color})

df_train['color'].unique()
# array(['green', 'red', 'purple'], dtype=object)

# testing data - yellow and black are unseen categorical featurs
x1_test = [35.0, 28.0, 30.0, 32.0, 46.0] 
x2_test = [0.44, 0.44, 0.6, 0.39, 0.39]
color_test =  ['green','red','purple','yellow','black']

df_test = pd.DataFrame({
'x1': x1_test,
'x2' :x2_test,
'color':color_test})

df_test['color'].unique()
# array(['green', 'red', 'purple', 'yellow', 'black'], dtype=object)


X = df_train[['x1', 'x2', 'color']]
y = df_train['y']

# I need to apply 3rd polynomial to x1 variable only. variable color is converted to dummy 
# variable
preprocessor = ColumnTransformer(
transformers=[
('encoder', OneHotEncoder(sparse=False), ['color']),
('transformer', PolynomialFeatures(degree=3, include_bias=False), ['x1']),
],
remainder='passthrough')

pipeline = Pipeline([
('preprocessor', preprocessor),
('regressor', LinearRegression(fit_intercept=True))])

pipeline.fit(X, y)

print(pipeline['regressor'].intercept_)
# -12.235254842701742

print(pipeline['regressor'].coef_)
# [ 1.12300403 -0.55836609 -0.56463793  0.12934888  0.19512496 -0.00390984
#  -0.20906133]

list_coeff = pipeline['regressor'].coef_ # get the coefficient
list_col = preprocessor.get_feature_names() # get name for each coefficient
dic = {list_col[i]: list_coeff[i] for i in range(len(list_col))} # create a dic for each 
# coefficient and its corresponding name
print(dic)

# {'encoder__x0_green': 1.123004029501841, 'encoder__x0_purple': -0.5583660948050801, 
#'encoder__x0_red': -0.5646379346959568, 
# 'transformer__x0': 0.12934888105186387, 'transformer__x0^2': 0.19512495572810412, 
#'transformer__x0^3': -0.003909843646823246, 
# 'x2': -0.20906132968981733}

# Also apply one hot encoder to testing data, so I can plug in the equation to predict Y in 
# testing data
columns_to_category = ['color']
df_test[columns_to_category] = df_test[columns_to_category].astype('category') 
df_test = pd.get_dummies(df_test, columns=columns_to_category) # One hot encoding the categories

df_test.columns
# Index(['x1', 'x2', 'color_black', 'color_green', 'color_purple', 'color_red',
#        'color_yellow'],
#       dtype='object')

# These are coefficient 
intercept = -12.235254842701742
poly3 = -0.00390984364682324
poly2 = 0.19512495572810412
poly1 = 0.12934888105186387
x2 = -0.20906132968981733
col_green = 1.123004029501841
col_purple = -0.5583660948050801
col_red = -0.5646379346959568

# Predict Y value from testing data. Problem is coefficient for color black and color yellow 
# are missing. Any solution to offer?
df_test['yhat'] = intercept + df_test['x1']**3*poly3 \
             + df_test['x1']**2*poly2  + df_test['x1']*poly1 \
             + df_test['x2'] * x2 \
             + df_test['color_black'] * col_blk \
             + df_test['color_green'] * col_green \
             + df_test['color_purple'] * col_purple \
             + df_test['color_red'] * col_red \
             + df_test['color_yellow'] * col_yellow

Answer 1

OneHotEncoder has parameters such as max_categories and handle_unknown . By setting handle_unknown='ignore' When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. You can see more information in docs [https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html]

Answer 2

When you're first fitting your encoder on the training set, save the categories OneHotEncoder produces.

oh = OneHotEncoder()
encoded = oh.fit_transform(categorical_attribute)
attribute_cats = oh.categories_

Then you can use those categories when transforming the test samples.

oh = OneHotEncoder(categories=attribute_cats)
test_encoded = oh.fit_transform(test.iloc[:3])

Categories, unseen in the testset, will have zeros in oh.categories_[0][i] columns.