R Parallel Package：clusterExport 到每个单独的核心，而不是一个大对象到所有核心

Question

I'm using the Parallel Package to save me some time in R. Usually I export whatever object I am trying to do calculations on with clusterExport and run the relevant function. However, this time I'm working with a very large object (10+GB) and even creating a small cluster uses a ton of RAM. The calculation I'm doing only requires a part of the 10 GB object for each copy of R. I'm wondering if there is a way to use clusterExport to only export the relevant half gigabyte or so to each copy of R. From what I can tell clusterExport always sends the entire object to each copy so if there is a work around to this please let me know.

You can see in this example below I store the data.frame x in each copy of R. My goal would be in this case to have 10 different datasets x1-x10 and load x1 to the first copy, x2 to the second, and so on. If there is another way to do this efficiently please let me know as well.

library('doParallel')
x <- data.frame(a=1:10,b=1:10)
test <- function(y) {
  return(x$a[y]+x$b[y])
}
cl <- makeCluster(10)
registerDoParallel(cl)
clusterExport(cl,list('test','x'))
test1 <- parLapply(cl,1:10,fun=test)
stopCluster(cl)

Answer 1

I came up with a solution to this after a few days. The solution I came up with was to save each portion of the data.frame in seperate csv files. Then within each virtual R session I read in the relevant csv file so each R session only contained the necessary data for their computations.