在sklearn中进行超参数调整后，查找模型的准确性，精确度和召回率

Question

I've a binary classification problem, for which I've chosen 3 algorithms, Logistic Regression, SVM and Adaboost. I'm using grid-search and k-fold cross validation on each of these to find the optimal set of hyper-parameters. Now, based on the accuracy, precision and recall I need to choose the best model. But the problem is I'm not able to find any suitable way to retrieve these information. My code is given below:

from sklearn.model_selection import GridSearchCV
from sklearn.metrics.scorer import make_scorer
from sklearn import cross_validation

# TODO: Initialize the classifier
clfr_A = LogisticRegression(random_state=128)
clfr_B = SVC(random_state=128)
clfr_C = AdaBoostClassifier(random_state=128)

lr_param_grid = {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000] }
svc_param_grid = {'C': [0.001, 0.01, 0.1, 1, 10], 'gamma' : [0.001, 0.01, 0.1, 1]}
adb_param_grid = {'n_estimators' : [50,100,150,200,250,500],'learning_rate' : [.5,.75,1.0,1.25,1.5,1.75,2.0]}

# TODO: Make an fbeta_score scoring object using make_scorer()
scorer = make_scorer(fbeta_score, beta = 0.5)

# TODO: Perform grid search on the classifier using 'scorer' as the scoring method using GridSearchCV()
clfrs = [clfr_A, clfr_B, clfr_C]
params = [lr_param_grid, svc_param_grid, adb_param_grid]

for clfr, param in zip(clfrs, params):
    grid_obj = GridSearchCV(clfr, param, cv=3, scoring=scorer, refit=True)
    grid_fit = grid_obj.fit(features_raw, target_raw)
    print grid_fit.best_estimator_
    print grid_fit.cv_results_

Problem is the cv_results_ gives out a lot of info but I'm not able to find anything relevant except mean_test_score . Moreover I don't see any accuracy, precision or recall related metric there.

I can think of one way to achieve it. I can change the for loop to look something like the following:

score_params = ["accuracy", "precision", "recall"]
for clfr, param in zip(clfrs, params):
    grid_obj = GridSearchCV(clfr, param, cv=3, scoring=scorer, refit=True)
    grid_fit = grid_obj.fit(features_raw, target_raw)
    best_clf = grid_fit.best_estimator_
    for score in score_params:
        print score,
        print " : ",
        print cross_val_score(best_clf, features_raw, target_raw, scoring=score, cv=3).mean()

But is there any better way of doing it? It seems I'm doing the operations multiple times for each model. Any help is appreciated.

Answer 1

GridSearchCV is doing what you gave. You gave the f_beta as scorer, so mean_test_score will return results of that f_beta for each parameter combination. If you want to access other metrics, you need to tell the GridSearchCV explicitly to do so.

GridSearchCV in newer versions of scikit-learn, supports multi-metric scoring. So you can pass multiple type of scorers in that. As per documentation :

scoring : string, callable, list/tuple, dict or None, default: None

... ...

For evaluating multiple metrics, either give a list of (unique) strings or a dict with names as keys and callables as values.

See this example here:

• http://scikit-learn.org/stable/auto_examples/model_selection/plot_multi_metric_evaluation.html#running-gridsearchcv-using-multiple-evaluation-metrics

And change your scoring param as:

scoring = {'Accuracy': 'accuracy', 
           'FBeta': make_scorer(fbeta_score, beta = 0.5),
           # ... Add others here as you want.
           }

But now when you do it, you need to change the refit param also. Since different metrics here will give different type of scores for the parameter combinations, so you need to decide which one to select when refitting the estimator. So choose one of the keys from the scoring dict for refit

for clfr, param in zip(clfrs, params):
    grid_obj = GridSearchCV(clfr, param, cv=3, scoring=scorer, refit='FBeta')
    ...
    ...