如何使用插入符号包解释模型输出的准确性

Question

I am using caret package to train a model and would like to get the accuracy of the model. A common way I heard is to use confusionMatrix. However, when I run my code below, the trained model gives me some accuracy values that are slightly different from what confusionMatrix() reports. So my question is what accuracy should I use? How to interpret the accuracy the model gives directly in the console?

ModelRF_ALL_b <- train(price~.,method="rf",data=datatraining_b)
ModelRF_ALL_b

The console reports the following

Random Forest 

8143 samples
   8 predictor
   2 classes: '0', '1' 

No pre-processing
Resampling: Bootstrapped (25 reps) 
Summary of sample sizes: 8143, 8143, 8143, 8143, 8143, 8143, ... 
Resampling results across tuning parameters:

  mtry  Accuracy   Kappa    
  2     0.9948108  0.9843501
  4     0.9945824  0.9836512
  7     0.9940732  0.9821099

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was mtry = 2.

I can also run confusionMatrix()

confusionMatrix(datatraining_b$price,
predict(ModelRF_ALL_b,datatraining_b))

It gives accuracy of 1.

Confusion Matrix and Statistics

      Reference
Prediction    0    1
     0 6414    0
     1    0 1729

           Accuracy : 1          
             95% CI : (0.9995, 1)
No Information Rate : 0.7877     
P-Value [Acc > NIR] : < 2.2e-16  

              Kappa : 1          
 Mcnemar's Test P-Value : NA         

        Sensitivity : 1.0000     
        Specificity : 1.0000     
     Pos Pred Value : 1.0000     
     Neg Pred Value : 1.0000     
         Prevalence : 0.7877     
     Detection Rate : 0.7877     
   Detection Prevalence : 0.7877     
  Balanced Accuracy : 1.0000     

   'Positive' Class : 0     

Answer 1

You can interpret these values as in-sample accuracies with and without resampling, respectively.

The package caret performs bootstrapped resampling with 25 repetition when you fit the model, which can be seen in your model output. So, the accuracy value is based on 25 x 8143 observations. In order to create the confusion matrix, you are using the final model (the one with mtry = 2) to predict the outcomes from the training sample, which has a length of 8143. Therefore, it is normal to have a slight difference in corresponding accuracies.

You need to be careful while assessing the goodness-of-fit because you are training and evaluating your model using the same dataset. No surprise that you get a large accuracy. It is always good to evaluate your final model with an unseen dataset to ensure its performance and detect possible over-fitting issues.