R 中 gbm 的缺失数据和分层 k 折交叉验证

Question

I have a relatively large data set on sales of homes in several markets in the US For each market, I want to build a Gradient Boosting regression model to predict the sale price. Most of my independent variables (features) have missing values which should be fine for a gbm in R.

The gbm algorithm in caret requires you to specify values of the hyperparameters ( n.trees , shrinkage , interaction.depth , n.minobsinnode , etc). I want to do a grid search in conjunction with cross validation to pick the best set of hyperparameters:

# -------- A function to drop variables that are more than 80% missing or have no variance
Drop_Useless_Vars <- function(datan) {
  n = nrow(datan)
  p = ncol(datan)
  na = is.na(datan)
  n_na = apply(X = na, MARGIN = 2, FUN = sum)
  n_unique = apply(X = datan, MARGIN = 2, function(x) length(na.omit((unique(x)))))
  return(as.data.frame(datan[, -which(n_na > 0.8*n | n_unique < 2)]))
}

# -------- load libraries
library(gbm)
library(caret)

# -------- prepare training scheme
control = trainControl(method = "cv", number = 5)

# -------- design the parameter tuning grid 
grid = expand.grid(n.trees = 10000, 
                   interaction.depth = seq(2, 10, 1), 
                   n.minobsinnode = c(3, 4, 5), 
                   shrinkage = c(0.1, 0.01, 0.001))

# -------- tune the parameters
tuner = train(log(saleprice) ~ ., data = Drop_Useless_Vars(df), method = "gbm", distribution = "gaussian",
              trControl = control, verbose = FALSE, tuneGrid = grid, metric = "RMSE")

# -------- get the best combo
n_trees = tuner$bestTune$n.trees
interaction_depth = tuner$bestTune$interaction.depth
shrinkage = tuner$bestTune$shrinkage
n_minobsinnode = tuner$bestTune$n.minobsinnode

The above code works fine except for some markets where missing values are much more frequent. I'm getting the error shown below:

Error in checkForRemoteErrors(val) : 
  4 nodes produced errors; first error: variable 26: assessor_full_baths has only missing values.

assessor_full_baths is one of the features in my model. So what's happening is that when the algorithm is sampling data to do the cross validation, one or more of the folds are having variables that are completely missing.

How can I stratify the sampling scheme used by caret ? That is, how can I force each fold to have the same distribution with respect to the missing values? Also, do you guys know how to make the gbm function ignore variables that are completely missing without us telling it which ones they are?

I will be grateful for any help you can provide.

Answer 1

I think you need to take a step back and think about how to properly handle the data and fit the model.

Your final modeling data set shouldn't have any missing values, let alone so many missing values that features are 100% missing in CV folds (!).

Instead, do a bit of data cleaning and feature engineering:

• you can add NA as a factor level to factor variables
• you can impute missing values in most other situations
• if a feature has high missingness, then you probably need to exclude it, or
• if it's not missing at random and there many be some information you can learn by the fact that it was missing / present, you might create one or a few indicator variables indicating if original feature was missing and/or if it contained a certain level of interest.

If you don't want to impute the missing data then you should use the other strategies mentioned. The missing node is not the same as encoding an NA level. The missing node just means that the tree will give the same prediction as prior to the split where data was missing (see R gbm handling of missing values ). You should never build a model on a feature with high missingness and it would not be a good practice at all to build a model on data with any missing values. Those should be cleaned when you prepare the data set.

Having said that, you can still impute MNAR data. There are a large number of strategies, going back to Heckman's and Rubin's work in the 70's. A lot of people use mice() and the drawn indicator method .

This may help: http://stefvanbuuren.nl/mi/docs/mnar.pdf

Answer 2

You can use caret::createFolds to create the CV folds yourself. You simply need to supply a different y variable than your outcome (ie- saleprice ). You'll want to define a new variable based on your stratification variables... see ?interaction . This can then be passed to caret::trainControl

For example:

library(caret)
y2 <- interaction(df$x1, df$x2)
cv_folds <- createFolds(y2, k= 5)
control = trainControl(index= cv_folds, method= "cv", number= 5)
...

Or you can avoid using caret and write your own stratified sampling code

That said, I agree with most everything @Hack-R said about the practical use of features with missing values and imputation.