rm（list = ls（））on 64GB然后错误：尝试决策树时无法分配大小为11.6 Gb的向量

Question

I'm trying to run a decision tree via caret package. I start my script fresh by removing everything from memory with rm(list = ls()) then I load my training data which is 3M rows and 522 features. R studio doesn't show the size in gb but presumably by the error message it's 11.6.

If I'm using 64gb R then is it expected I see this error? Is there any way around it without resorting to training on smaller data?

rm(list = ls())
library(tidyverse)
library(caret)
library(xgboost)

# read in data
training_data <- readRDS("/home/myname/training_data.rds")

R studio environment pane currently shows one object, training data with the dims mentioned above.

### Modelling
# tuning & parameters
set.seed(123)
train_control <- trainControl(
  method = "cv",
  number = 5,
  classProbs = TRUE, # IMPORTANT!
  verboseIter = TRUE,
  allowParallel = TRUE
)

# Fit a decision tree (minus cad field)
print("begin decision tree regular")
mod_decitiontree <- train(
  cluster ~.,
  tuneLength = 5,
  data = select(training_data, -c(cad, id)), # a data frame
  method = "rpart",
  trControl = train_control,
  na.action = na.pass
)

Loading required package: rpart
Error: cannot allocate vector of size 11.6 Gb

I could ask our admin to increase my RAM but before doing that want to make sure I'm not missing something. Don't I have lot's of RAM available if I'm on 64 GB?

Do I have any options? I tried making my data frame a matrix and passing that to caret instead but it threw an error. Is passing a matrix instead a worthwhile endevour?

Answer 1

Here is your error message reproduced:

cannot allocate vector of size 11.6 Gb when trying a decision tree

This means that the specific failure happened when R requested another 11.6 GB of memory, and was unable to do so. However, the random forest calculation itself may require many such allocations, and, most likely, the remainder of free RAM was already being used.

I don't know the details of your calculation, but I would say that even running random forests on a 1GB data set is already very large. My advice would be to find a way to take a statistically accurate sub sample of your data set such that you don't need such large numbers of RAM.