R中xgboost回归的置信区间

Question

I am currently working on a dataset that contains 4 categorical input variables and one numeric output.

I created a model, using the xgboost package in R. But I cannot find a way to compute a CI.

How can I compute the confidence interval for my predictions? I found this answer to a classification problem , but I do not understand it properly. Could someone explain it in more depth for my problem?

Answer 1

From what I can tell, there isnt a direct way to compute this using the xgboost package.

The linked article you gave gives a framework for how you could go about do it. It references doing "bagging" , which basically means creating the same model many times (that has randomness in it). For xgboost, if you were to set the colsample_bytree (what random selection of columns to use in each tree) to < 1 and subsample (what random percent of rows to use in each tree) < 1 , then this will introduce a "random element" to the model.

If you set the above variables to less than 1, you would have a model with a random element. If you were to run this model 100 different times, each time with a different seed value, you would end up with 100 unique xgboost models technically, with 100 different predictions for each observation. Using these 100 predictions, you could come up with a custom confidence interval using the mean and standard deviation of the 100 predictions.

I cant vouch for how effective or reliable these custom confidence intervals would be, but if you wanted to follow the example in the linked article this how you would do it, and this is the explanation of what they were talking about.

Here is some sample code for doing this, assuming you have 500 observations:

##make an empty data frame with a column per bagging run
predictions <- data.frame(matrix(0,500,100))

library(xgboost)

##come up with 100 unique seed values that you can reproduce
set.seed(123)
seeds <- runif(100,1,100000)

for (i in 1:ncol(predictions){

set.seed(seeds[i])
xgb_model <- xgboost(data = train,
                     label = y,
                     objective = "reg:linear",
                     eval_metric = "rmse",
                     subsample = .8,
                     colsample_bytree = .8
                     )

predictions[,i] <- predict(xgb_model,newdata = test)

}

Answer 2

A great option to get the quantiles from a xgboost regression is described in this blog post. I believe this is a more elegant solution than the other method suggest in the linked question (for regression).

https://www.bigdatarepublic.nl/regression-prediction-intervals-with-xgboost/

Basically your problem can be described as followed (from the blog):

In the case that the quantile value q is relatively far apart from the observed values within the partition, then because of the Gradient and Hessian both being constant for large difference x_i-q, the score stays zero and no split occurs.

Then the following solution is suggested:

An interesting solution is to force a split by adding randomization to the Gradient. When the differences between the observations x_i and the old quantile estimates q within partition are large, this randomization will force a random split of this volume.