How to calculate the 95% confidence interval from a model trained using CARET in R?

Question

I have built different regression models using the R package caret . How can we calculate the 95% confidence interval for the predictions? I have followed the discussion noted here , however, it is not working.

rm(list = ls())
library(caret)

data("mtcars")
Train_data = mtcars[1:26, -c(8,9)]
Test_data = mtcars[27:32, -c(8,9)]


set.seed(100)
model_pls <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 5, 
  method = "pls", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'), 
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3, 
    savePredictions = "final"
  )
)

model_rf <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 5, 
  method = "ranger", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'), 
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3, 
    savePredictions = "final"
  )
)

model_svmr <- train(
  hp ~ ., 
  data = Train_data, 
  tuneLength = 8, 
  method = "svmRadial", 
  metric = "RMSE", 
  preProcess = c('center', 'scale'),
  trControl = trainControl(
    method = "repeatedcv", 
    number = 5, 
    repeats = 3,
  )
)

# This does not generate confidence interval
PLS.pred = predict(model_pls, subset(Test_data, select = -hp))  
RF.pred = predict(model_rf, subset(Test_data, select = -hp)) 
RF.svm = predict(model_svmr , subset(Test_data, select = -hp)) 


# This is not working
predict(model_pls$finalModel, subset(Test_data, select = -hp), interval = "confidence")
predict(model_rf$finalModel, subset(Test_data, select = -hp), interval = "confidence")
predict(model_svmr$finalModel, subset(Test_data, select = -hp), interval = "confidence")

Following Michael Matta's suggestion, I have tried the following code, however, it is not working as expected.

confint(model_pls, level = 0.95)
# Error in UseMethod("vcov"): no applicable method for 'vcov'

predict(model_pls, subset(Test_data, select = -hp), interval = "confidence")
# 64.47807  57.97479 151.59713 130.24356 183.20296  88.50035
# This does not show the CI.

Answer 1

Confidence intervals come either from a known distribution and the following statistics or are constructed using resampling. RBF SVM, random forrests etc. do not have known distributions, ie, they cannot yield confidence interval on anything as they are the same way as linear models (lm).

The way to obtain confidence intervals from such models is resampling train/test datasets, re-training, collect the values you need (eg, using a for-loop). Then, from such collected data, estimate expected value confidence interval through known distribution of the mean.

---

The following pseudocode should work for nearly any score you would like (accuracy, RMSE, ...; for comment, see below):

predictionsTrainAll <- c()
predictionsTestAll <- c() 
scoresTrain <- c()
scoresTest <- c()

for( i in 1:1000){
    d <- shuffle the original dataset,
    training <- draw training dataset from d,
    testing  <- draw testing datassetfrom d (such that training and testing do not have any intersection),
    
    model <- train a model on training data,
    predictionsTrain <- make predictions for training data,
    predictionsTest  <- make predictions for testing data,
    scoreTrain <- evaulate model and obtain any score you like on train,
    scoreTest  <- evaluate model and obtain any score you like on test,
    
    predictionsTrainAll <- append(predictionsTrainAll, predictionsTrain)
    predictionsTestAll <- append(predictionsTestAll, predictionsTest)
    scoresTrain <- append(scoresTrain, scoreTrain)
    scoresTest  <- append(scoresTest, scoreTest)
}

Now, we can estimate the expected value for scoresTrain and scoresTest. Due to the central limit theorem, we can assume the expected value has normal distribution (or t-distribution as we have finite samples here). We can use:

# scores should be /somehow/ normally distributed (symmetric by mean, meadian close to the mean)
hist(predictionsTrainAll)
hist(predictionsTestAll)
hist(scoresTrain)    
hist(scoresTest)     

# if the histogram are /somehow/ normal:
t.test(predictionsTrainAll)
t.test(predictionsTestAll)
t.test(scoresTrain)
t.test(scoresTest) 

that will calculate the 95% confidence interval of the expected value (true mean) of the predicted values and any scores you want. But beware, if the histograms are skewed, the estimation of the mean can be flawed and yield wrong confidence intervals.

Example case for a binary classifier: The estimated true mean for the predictions is 0 with 95 % CI = [-0.32, 0.32], since the model predicts zeros. However, the predictions can be are only inbetween [0; 1] and thus the negative part of the CI does not make sense. Such CI is a consequence of symmetricity implied by normal/t-distribution. This can happen when the histogram of the examined scores/predictions are not normally distributed.