GridSearchCV 没有提高我的测试准确性

Question

I am making multiple classifier models and the test accuracy for all of them is 0.508.

I find it weird that multiple models have the same accuracy. The models I used are Logistic Regressor,DesicionTreeClassifier, MLPClassifier, RandomForestClassifier, BaggingClassifier, AdaBoostClassifier, XGBClassifier, SVC, and VotingClassifier.

After using GridSearchCV to improve the models, all of their test accuracy scores improved. But the test accuracy scores did not change.

I wish I could say I changed something, but I don't know why the test scores did not change. After using gridsearch, I expected the test scores to improve but it didn't

Answer 1

I would like to confirm, you mean your training scores improve but you testing scores did not change? If yes, there are a lot of possibility behind this.

• You might want to reconfigure and add your hyper parameter range for example if using KNN you can increase the number of k or by adding more distance metric calculation
• If you want to you can change the hyper parameter optimization technique like randomized search or bayesian search
• I don't have any information about your data but sometimes turn on or turn off the shuffle mode when splitting can affect the scores for instance if you have time series data you have not to shuffle the dataset

Answer 2

Looking at your accuracy, first of all I would say: are you performing a binary classification task? Because if it is the case, your models are almost not better than random on the test set, which may suggest that something is wrong with your training.

Otherwise, GridSearchCV , like RandomSearchCV and other hyperparameters optimization techniques try to find optimal parameters among a range that you define . If, after optimization, your optimal parameter has the value of one bound of your range, it may suggest that you need to explore beyond this bound, that is to say set another range on purpose and run the optimization again.

By the way, I don't know the size of your dataset but if it is big I would recommend you to use RandomSearchCV instead of GridSearchCV . As it is not exhaustive, it takes less time and gives results that are (nearly) optimized.

Answer 3

There can be several reasons why the test accuracy didn't change after using GridSearchCV:

The best parameters found by GridSearchCV might not be optimal for the test data.

The test data may have a different distribution than the training data, leading to low test accuracy.

The models might be overfitting to the training data and not generalizing well to the test data.

The test data size might be small, leading to high variance in test accuracy scores.

The problem itself might be challenging, and a test accuracy of 0.508 might be the best that can be achieved with the current models and data.

It would be useful to have more information about the data, the problem, and the experimental setup to diagnose the issue further.