为什么（rf）clf feature_importance重视所有值相同的功能？

Question

I am comparing multi-classification with Random Forests and CART in scikit-learn.

Two of my features (feature 4 and feature 6) are irrelevant for the classification because all the values are the same. But output of the feature_importances of the RandomForestClassifier is the following:

Feature ranking:

1. feature 3 (0.437165)
2. feature 2 (0.216415)
3. feature 6 (0.102238)
4. feature 5 (0.084897)
5. feature 1 (0.064624)
6. feature 4 (0.059332)
7. feature 0 (0.035328)

CART feature_importance output:

Feature ranking:

1. feature 3 (0.954666)
2. feature 6 (0.014117)
3. feature 0 (0.011529)
4. feature 1 (0.010586)
5. feature 2 (0.006785)
6. feature 4 (0.002204)
7. feature 5 (0.000112)

In every row, feature 4 has the same value. Same is for feature 6.

Here is the code

Random Forest

importances = rfclf.feature_importances_
std = np.std([tree.feature_importances_ for tree in rfclf.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(x.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

CART

importances = clf.feature_importances_
std = np.std([tree.feature_importances_ for tree in rfclf.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(x.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

I would except that the importance is like:

6. feature 6 (0.000000)
7. feature 4 (0.000000)

When I just don't use that two features, my models overfit.

Answer 1

You need to set a limit to the depth of your trees. I recommend doing a gridsearch over min_samples_leaf=[0.001, 0.1] - trying between 0.1% to 10% in each leaf.

Any kind of feature importance calculation must be done on a robust model to be meaningful.