Prediction Intervals for R tidymodels Stacked model from stacks()

Question

Is it possible to calculate prediction intervals from a tidymodels stacked model?

Working through the example from the stacks() package here yields the stacked frog model (which can be downloaded here for reprex) and the testing data:

data("tree_frogs")
tree_frogs <- tree_frogs %>%
  filter(!is.na(latency)) %>%
  select(-c(clutch, hatched))

set.seed(1)
tree_frogs_split <- initial_split(tree_frogs)
tree_frogs_train <- training(tree_frogs_split)
tree_frogs_test  <- testing(tree_frogs_split)

I tried to run something like this: pi <- predict(tree_frogs_model_st, tree_frogs_test, type = "pred_int")

but this gives an error: Error in UseMethod("stack_predict"): no applicable method for 'stack_predict' applied to an object of class "NULL"

Reading the documentation of stacks() I also tried passing "pred_int" in the opts list: pi <- predict(tree_frogs_model_st, tree_frogs_test, opts = list(type = "pred_int"))

but this just gives: opts is only used with type = raw and was ignored.

For reference, I am trying to do a similar thing that is done in Ch.19 of Tidy Modeling with R book

lm_fit <- fit(lm_wflow, data = Chicago_train)
 predict(lm_fit, Chicago_test, type = "pred_int")

which seems to work fine for a single model fit like lm_fit , but apparently not for a stacked model?

Am I missing something? Is it not possible to calculate prediction intervals for stacked models for some reason?

Answer 1

This is very difficult to do.

Even if gl.net produced a prediction interval, it would be a significant underestimate since it doesn't know anything about the error in each of the ensemble members.

We would have to get the standard error of prediction from all of the models to compute it for the stacking model. A lot of these models don't/can't generate that standard error.

The alternative is the use bootstrapping to get the interval but you would have to bootstrap each model a large number of times to get the overall prediction interval.