分别保存模拟数据集（速度+内存限制）

Question

I am working with a very large number (say two million) simulated datasets that are created in a for-loop and listed in groups of 2000. I would like to save all 1000 lists of 2000 datasets somewhere, such that I can perform any analysis without having to generate data again. Saving all two million datasets in a nested list exceeds memory, so that is not an option. Therefore, I tried to save them per sublist in a workspace:

# Generate data
data_list <- vector("list", 2000)

for(i in 1:1000){
    for(j in 1:2000){
        dataA <- cbind(rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j))
        dataB <- cbind(rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j))
        data_list[[j]] <- dataA-dataB
    }

# Write to workspace
assign(paste("Data",i,sep=""), data_list)

# Add to existing workspace and remove object (to save memory).
if(file.exists("Workspaces.RData")){  
    old.objects <- load("Workspaces.RData")
    save(list=c(old.objects, paste("Data",i,sep="")),file="Workspaces.RData") 
    rm(list=c(old.objects,paste("Data",i,sep="")))

# Or create new workspace if it does not exist
}else{
    save(list=paste("Data",i,sep=""), file="Workspaces.RData")
    rm(list=paste("Data",i,sep=""))}
}

This is a very slow solution for the number and sizes of datasets I am working with, so I was wondering whether anyone has a better solution to store and load generated datasets.

Thanks in advance!

Answer 1

As mentioned by F.Privé, if you need to save those files, better use saveRDS. In that case you are not doing redundant saving and loading.

jj <- 1:2000
for(i in 1:10){
  for(j in jj){
    dataA <- cbind(rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j))
    dataB <- cbind(rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j),rnorm(j))
    data_list[[j]] <- dataA-dataB
  }
  saveRDS(data_list, paste0("Data", i, '.rds'))
}

As of this particular data simulation, I would try to avoid loops. Generating all the data at once (or at least in parts) and then storing to data.frame with index column. Something like:

dataA <- replicate(8, rnorm(sum(jj)))
dataB <- replicate(8, rnorm(sum(jj)))
data_list <- dataA - dataB
data <- as.data.frame(data_list)
data[, "ind"] <- rep(jj, times = jj)

But as I assume this is not your real data simulation, it is crucial to understand why you are simulating 1k list of 2k data sets? Do they all need to be in separated lists? All are simulated equally? and so on...