如何在sklearn中使用DictVectorizer后获得分类特征的重要性

Question

I'm using sklearn.ensemble.GradientBoostingRegressor to train a model.

My data set includes heterogeneous variables, both numeric and categroical variables. Since sklearn does not support categroical variables, I use DictVectorizer to convert these categorical features before inputting into Regressor . Here is a piece of my code:

# process numeric and categorical variables separately
lsNumericColumns = []
lsCategoricalColumns = []
for col in dfTrainingSet.columns:
    if (dfTrainingSet[col].dtype == np.object):
        lsCategoricalColumns.append(col)
    else:
        lsNumericColumns.append(col)

# numeric columns
dfNumVariables = dfTrainingSet.loc[:, lsNumericColumns]
dfNumVariables.fillna(0, inplace=True)
arrNumVariables = dfNumVariables.as_matrix()

# categorical columns
dfCateVariables = dfTrainingSet.loc[:,lsCategoricalColumns]
dfCateVariables.fillna('NA', inplace=True)
vectorizer =  DictVectorizer(sparse=False)
arrCateFeatures = vectorizer.fit_transform(dfCateVariables.T.to_dict().values())

# setup training set
arrX = np.concatenate((arrNumVariables,arrCateFeatures), axis=1)
arrY = dfData['Y'].values

Then, train the model and output the feature importance:

# setup regressor
params = {'n_estimators':500, 'max_depth':10, 'min_samples_split':50, \
          'min_samples_leaf':50, 'learning_rate':0.05, 'loss':'lad', \
          'subsample':1.0, 'max_features':"auto"} 
gbr = GradientBoostingRegressor(**params) 

# fit
print('start to train model ...') 
gbr.fit(arrX, arrY) 
print('finish training model.')

print(gbr.feature_importances_)

This will give me a list of (feature_index, feature_importance) tuples. However, I find that this feature index is not the original feature index as one categorical column can be converted into several columns.

I understand I can get vectorized feature name from DictVectorizer, but how can I find out the importance of original features?

could I just sum up all the importance of vectorized features which corresponds to same original feature to get the importance of original feature?

Answer 1

You can get the feature importances for the one-hot features with

zip(vectorizer.get_feature_names(), gbr.feature_importances_)

This gives a list of (feature, importance) pairs, where features are of the form 'name=value' for categoricals and just name for originally numerical features. The order of appearance in get_feature_names output is guaranteed to match the order in transform or fit_transform output.

To be honest, I'm not sure about the feature importances for the original categoricals; I'd try taking the mean rather than the sum.