R-向量內存已耗盡（是否達到極限？）嵌套循環的內存問題？

Question

我目前正在嘗試編寫R腳本來導入我創建的與數據集相關的各種文件。 這涉及根據我如何組織文件的目錄和名稱來使用多個嵌套的for循環讀取許多.txt文件。

我可以運行最里面的循環（稍微慢一點）。 但是，嘗試運行第二個循環或任何其他循環會產生以下錯誤：

Error: vector memory exhausted (limit reached?)

我相信這可能與R如何處理內存有關？ 我正在Rstuidio中運行R。 我也嘗試過這里沒有運氣的解決方案

'R
 R version 3.5.1 (2018-07-02) -- "Feather Spray"
 Platform: x86_64-apple-darwin15.6.0 (64-bit)

下面的代碼

subjects <- 72
loop1_names <- as.character(list('a','b','c'))
loop2_names <- as.character(list('one','two','three'))
loop3_names <- as.character(list('N1','N2'))
loop4_names<- as.character(list('choice1','choice2','choice3'))
i<-1;j<-1;

loop3.subset<- data.frame
for(k in 1:length(loop3_names)){

  loop4.subset<- data.frame()#Data frame for handling each set of loop 4 values
  for(l in 1:length(loop4_names)){

            #Code for extracting the variables for each measure

            measures.path <- file.path(results_fldr, 'amp_measures',loop1_names[i],loop2_names[j],'mont',loop3_names[k])
            measures.data <- read.table(file.path(measures.path, paste(paste(loop1_names[i],loop2_names[j],loop3_names[k],loop4_names[l],sep = '_'),'.txt',sep = '')), header = T, nrows = subjects)

            #Get rid of the IDs, we'll add those back in later
            col_idx_ID <- grep('ID', names(measures.data))
            measures.data <- as.data.frame(measures.data[,-col_idx_ID])# make sure when trimming to keep the measures as a data frame
            names(measures.data) <- c(paste(loop1_names[i],loop2_names[j],loop3_names[k],loop4_names[l],sep = '_'))#Add a label to the data

            #Now combine this data with the other data in the loop4 subset data frame
            if(l == 1){
              loop4.subset <- measures.data
            } else {
              loop4.subset <- merge(erp.subset,measures.data)
            }

          }#End l/loop 4
          if(k == 1){
            loop3.subset <- loop4.subset
          } else {
            freq.subset <- merge(loop3.subset,loop4.subset)
          }

        }#End k/loop 3

Answer 1

通常，我建議您僅將部分數據讀取到內存，然后將部分合並到磁盤。 在下面的示例中，我當然沒有運行，因為我沒有您的文件。 我在每個i，j循環之后寫入磁盤，然后完成之后有9個文件。 現在，您將這6個文件合並到另一個循環中。 如果仍然存在內存問題，請首先執行“ j”合並並將每個文件寫入3個“ i”文件，將其分解為另外2個文件。 然后，如果您無法合並這些文件，則可能會遇到根本問題，即計算機上的內存不足。

subjects <- 72
loop1_names <- as.character(list('a','b','c'))
loop2_names <- as.character(list('one','two','three'))
loop3_names <- as.character(list('N1','N2'))
loop4_names<- as.character(list('choice1','choice2','choice3'))

for(i in 1:length(loop1_names)) {
    for(j in 1:length(loop2_names)) {
        loop3.subset<- data.frame
        for(k in 1:length(loop3_names)){

            loop4.subset<- data.frame()
            for(l in 1:length(loop4_names)){

                ##Code for extracting the variables for each measure

                measures.path <- file.path(results_fldr,
                                           'amp_measures',
                                           loop1_names[i],
                                           loop2_names[j],
                                           'mont',
                                           loop3_names[k])
                measures.data <- read.table(file.path(measures.path, paste(paste(loop1_names[i],
                                                                                 loop2_names[j],
                                                                                 loop3_names[k],
                                                                                 loop4_names[l],
                                                                                 sep = '_'),'.txt',sep = '')),
                                            header = T, nrows = subjects)

                ##Get rid of the IDs, we'll add those back in later
                col_idx_ID <- grep('ID', names(measures.data))
                measures.data <- as.data.frame(measures.data[,-col_idx_ID])
                names(measures.data) <- c(paste(loop1_names[i],
                                                loop2_names[j],
                                                loop3_names[k],
                                                loop4_names[l],
                                                sep = '_'))

                ## Now combine this data with the other data in the loop4 subset data frame
                if(l == 1){
                    loop4.subset <- measures.data
                } else {
                    loop4.subset <- merge(erp.subset,measures.data)
                }

            }#End l/loop 4
            if(k == 1){
                loop3.subset <- loop4.subset
            } else {
                freq.subset <- merge(loop3.subset,loop4.subset)
            }
        }#End k/loop 3
        write.table(freq.subset, paste0(i, "_", j, ".txt"))
    }
}

## Now you have 6 files to read in a merge.
## Something like this:

df <- NULL
for(i in 1:length(loop1_names)) {
    for(j in 1:length(loop2_names)) {
        df1 <- read.table(paste0(i, "_", j, ".txt"))
        df <- merge(df, df1)
    }
}