如何在 Sci-kit 中对多输出回归器使用交叉验证？

Question

first my setup: X is my feature table. It has 150 000 features and 96 samples. So 150 000 columns and 96 rows.

y is my target table. It has 4 labels and of course 96 samples. So 4x96 (columns x rows).

After splitting into train and test data I'm using MLPRegressor. Based on the documentation of Sci-kit it is an native multioutput regressor. So I can use it to predict my four desired output values with a new sample of 150 000 features . My code:

mlp = MLPRegressor(hidden_layer_sizes=(2000, 2000), solver= 'lbfgs', max_iter=100)
mlp.fit(X_train,y_train)

And then I'm using cross validation.

cross_validation.cross_val_score(mlp, X, y, scoring='r2')

The output is a list with 3 entries (parameter cv=3). I don't really get how my 4 labels get represented by these 3 values. I expected something in a format like this: label 1: 3 entries, label 2: 3 entries and the same with label 3 and 4. So I'm getting the R^2-Value for all my labels three times for different splittings of test and train data.

Am I missing something? Do I need to use Multioutputregressor? (See doc here)

And Here the documentation of cross validation.

Thanks.

Answer 1

First thing is if you are actually using cross_validation.cross_val_score() , then you should replace that with model_selection.cross_val_score() . Module cross_validation has been deprecated and removed from latest version of scikit.

Now coming to why you are only getting a single score for all your outputs and not individual entries is because thats how the default value of scorer is set.

You have used scoring 'r2' which is documented here . In that, there is an option to change the result if the input is multi-output (as your case) by using the

multioutput : Defines aggregating of multiple output scores. Array-like value defines weights used to average scores. Default is “uniform_average”.

'raw_values' : Returns a full set of scores in case of multioutput input.

'uniform_average' : Scores of all outputs are averaged with uniform weight.

'variance_weighted' : Scores of all outputs are averaged, weighted by the variances of each individual output.

You see that the default value is 'uniform_average' , which just averages all the outputs to get a single value, which is what you are getting.