R multiclass/multinomial classification ROC using multiclass.roc (Package ‘pROC’)

Question

I am having difficulties understanding how the multiclass.roc parameters should look like. Here a snapshot of my data:

> head(testing.logist$cut.rank)
[1] 3 3 3 3 1 3
Levels: 1 2 3
> head(mnm.predict.test.probs)
              1            2          3
9  1.013755e-04 3.713862e-02 0.96276001
10 1.904435e-11 3.153587e-02 0.96846413
12 6.445101e-23 1.119782e-11 1.00000000
13 1.238355e-04 2.882145e-02 0.97105472
22 9.027254e-01 7.259787e-07 0.09727389
26 1.365667e-01 4.034372e-01 0.45999610
> 

I tried calling multiclass.roc with:

multiclass.roc(
        response=testing.logist$cut.rank,
        predictor=mnm.predict.test.probs,
        formula=response~predictor
        )

but naturally I get an error:

Error in roc.default(response, predictor, levels = X, percent = percent,  : 
  Predictor must be numeric or ordered.

When it's a binary classification problem I know that 'predictor' should contain probabilities (one per observation). However, in my case, I have 3 classes, so my predictor is a list of rows that each have 3 columns (or a sublist of 3 values) correspond to the probability for each class. Does anyone know how should my 'predictor' should look like rather than what it's currently look like ?

Answer 1

The pROC package is not really designed to handle this case where you get multiple predictions (as probabilities for each class). Typically you would assess your P(class = 1)

multiclass.roc(
    response=testing.logist$cut.rank,
    predictor=mnm.predict.test.probs[,1])

And then do it again with P(class = 2) and P(class = 3). Or better, determine the most likely class:

predicted.class <- apply(mnm.predict.test.probs, 1, which.max)
multiclass.roc(
    response=testing.logist$cut.rank,
    predictor=predicted.class)

Consider multiclass.roc as a toy that can sometimes be helpful but most likely won't really fit your needs.