sci-kit学习中关于SVC概率输出的网格搜索交叉验证

Question

I'd like to run a grid search cross-validation on the probability outputs of the SVC classifier. In particular I'd like to minimize the negative log likelihood. From the documentation it seems that GridSearchCV calls the predict() method of the estimator it is passed and the predict() method of SVC returns class predictions not probabilities ( predict_proba() returns class probabilities).

1) Do I need to subclass SVC and give it a predict() method that returns probabilities rather than classes to accomplish my log likelihood cross validation? I guess I then need to write my own score_func or loss_func ?

2) Is cross-validating on this negative log likelihood dumb? I'm doing it b/c the dataset is: a) imbalanced 5:1 and b) not at all separable ie even the "worst" observations have a > 50% chance of being in the "good" class. (Will probably also post this 2nd question on stats q&a)

Answer 1

1. Yes, you would, on both accounts.

    class ProbSVC(SVC): def predict(self, X): return super(ProbSVC, self).predict_proba(X) 

2. I'm not sure if this would work since the majority class may still dominate the log-likelihood scores and the final estimator might still produce >.5 positive for samples of the minority class. I'm not sure, though, so please post this to stats.

Answer 2

With the new scorer interface in the development version of sklearn, you do not need subclassing. You only need to define a scoring object as described in the docs Basically you need to do log_loss_score = Scorer(neg_log_loss, needs_threshold=True) This will possibly fallback to "decision_function", though.

You could also define a new scorer class that calls predict_proba on the estimator to ensure that it gets normalized probabilities.

Also, a pull request for log-loss would be welcome :)