Decision Tree Sklearn -Depth Of tree and accuracy

Question

I am applying Decision Tree to a data set, using sklearn

In Sklearn there is a parameter to select the depth of the tree - dtree = DecisionTreeClassifier(max_depth=10).

My question is how the max_depth parameter helps on the model. how does high/low max_depth help in predicting the test data more accurately?

Answer 1

max_depth is what the name suggests: The maximum depth that you allow the tree to grow to. The deeper you allow, the more complex your model will become.

For training error, it is easy to see what will happen. If you increase max_depth , training error will always go down (or at least not go up).

For testing error, it gets less obvious. If you set max_depth too high , then the decision tree might simply overfit the training data without capturing useful patterns as we would like; this will cause testing error to increase. But if you set it too low , that is not good as well; then you might be giving the decision tree too little flexibility to capture the patterns and interactions in the training data. This will also cause the testing error to increase.

There is a nice golden spot in between the extremes of too-high and too-low. Usually, the modeller would consider the max_depth as a hyper-parameter, and use some sort of grid/random search with cross-validation to find a good number for max_depth .

Answer 2

The ideal parameters really depend on your data. Too low will lead to underfitting, too high will lead to overfitting. It's best to experiment to find the sweet spot. A decision tree will always have a tendency to simply memorize your dataset, though.

Also, you may want to check out GridSearchCV . It automates the experimentation process and finds the ideal parameters.