精度得分错误：分类指标无法处理多类和连续目标的混合

Question

I am currently in the process of trying to optimize a model for a certain performance metric using Hyperparameter tuning. I did not receive the above error until I tried to tune my model specifically for the performance_score metric, when I just used my model as a normal predictor everything ran very smoothly, even when I added hyperparameter tuning it was fine. Only when I tried to optimize it to a certain metric did it throw this error. This is the relevant code:

scorers = {
    
    'precision_score': make_scorer(precision_score),
    'recall_score': make_scorer(recall_score),
    'accuracy_score': make_scorer(accuracy_score)
}
rf = RandomForestRegressor()
cv=KFold(n_splits=5,random_state=1,shuffle=False)
rf_rsearch = RandomizedSearchCV(estimator = rf, param_distributions = random_hypparams, scoring=scorers,refit='precision_score', return_train_score=True, n_iter = 50, cv = cv , verbose=2, n_jobs = -1)
rf_rsearch.fit(OS_x, OS_y)

In this code:

• random_hyperparams: is just a grid of randomized hyperparameters to be tested to find the best set.
• OS_x, OS_y are the x and y training set oversampled using SMC.With the respective shapes: (1290, 33) (1290,)

The error seems to be occurring in the final line of the chunk of code displayed. Any help would be really great as I cannot find anything online which is similar to this problem.

Answer 1

The error message indicates that the precision_score you are using does not support scoring the types of outputs/targets you are passing. Does this model make multiclass or continuous predictions?

If this is a multiclass predictor, you should be able to optimize according to precision_score, but you need to define an average for the multiclass targets by passing one of 'micro', 'macro', 'samples', or 'weighted' to precision_score's average parameter as the default='binary (see documentation) . The accepted answer to this StackOverflow post should help you do that with make_scorer.

If your model is making continuous predictions you will need to select a scorer that supports continuous output/target evaluation. The metrics under "Regression" in the scikit-learn documentation may help.