按代码列排序多个data.tables在R中进入相同数量的data.tables而不绑定data.tables（由于内存限制）

Question

I have many CSV's containing a huge amount of data that is unsorted by code across all CSV's in the set. I'd like to sort the codes across the whole set saving groups of codes to CSV's together, keeping the same number of CSV's as before when they were unsorted. I can't bind them together, sort, and split (as I usually would) because I have to keep the CSV's separated due to memory limits. My real dataset is billions of lines split across hundreds of CSV's like this.

For example, if after fread each of the data table examples below:

Reproducible data:

###Really I would fread() each of these, but reproducible here
data1 <- data.table(code=rep(c(1:2000),times=500),
                   data1=rep(c(10001:12000),times=500), 
                   data2=rep(c(20001:22000),times=500))
data2 <- data.table(code=rep(c(1:2000),times=500),
                    data1=rep(c(10001:12000),times=500), 
                    data2=rep(c(20001:22000),times=500))
data3 <- data.table(code=rep(c(1:2000),times=500),
                    data1=rep(c(10001:12000),times=500), 
                    data2=rep(c(20001:22000),times=500))
data4 <- data.table(code=rep(c(1:2000),times=500),
                    data1=rep(c(10001:12000),times=500), 
                    data2=rep(c(20001:22000),times=500))

I'd like to sort by the code for each of data (there is a variable number in reality) and save as the same number of csv's

The below is an example of the above data in the format I'd like. So there are codes 1-2000 on the original data.tables, here the codes are split so codes 1:500 is on desired1, codes 501:1000 are on desired2, codes 1001:1500 are on desired3, and codes 1501:2000 are on desired4.

Reproducible desired data:

###I'd use fwrite to save each one of these as a csv to file

desired1 <- data.table(code=rep(c(1:500),times=2000),
                                data1=rep(c(10001:10500),times=2000), 
                                data2=rep(c(20001:20500),times=2000))
desired2 <- data.table(code=rep(c(501:1000),times=2000),
                                data1=rep(c(10501:11000),times=2000), 
                                data2=rep(c(20501:21000),times=2000))
desired3 <- data.table(code=rep(c(1001:1500),times=2000),
                                data1=rep(c(11001:11500),times=2000), 
                                data2=rep(c(21001:21500),times=2000))
desired4 <- data.table(code=rep(c(1501:2000),times=2000),
                                data1=rep(c(11501:12000),times=2000), 
                                data2=rep(c(21501:22000),times=2000))

In reality I have 500 or more CSV's. What is the fastest way to sort them then save all of the same code to the same csv, while still splitting across the same number of csv's as the original unsorted files? Thanks in advance!

Answer 1

A for loop that sequentially rbind would be memory efficient

out <- data1[code %in% 1:500]
for(i in 2:4) out <- rbind(out, get(paste0('data', i))[code %in% 1:500])
identical(out, desired1) 
#[1] TRUE 

Answer 2

mm = function(x){
  a = table(x)
  rep(1:unique(a),length(a))
}

Map(function(x,y)set(x,j="code",value=mm(x[,code])+y),mget(ls(pattern = "data")),c(0,500,1000,1500))

$data4
         code data1 data2
      1: 1501 10001 20001
      2: 1502 10002 20002
      3: 1503 10003 20003
      4: 1504 10004 20004
      5: 1505 10005 20005
     ---                 
 999996: 1996 11996 21996
 999997: 1997 11997 21997
 999998: 1998 11998 21998
 999999: 1999 11999 21999
1000000: 2000 12000 22000

This changes the original data as it calls by reference. ie try calling data2 you will see that it has changed. If you do not want this behavior, you might consider using the function copy ie set(copy(x),....