以更快的方式读取 R 中的大型 RDS 文件

Question

I have a large RDS file to read in R. However, it takes quite some time to read the file.

Is there a way to speed up the reading? I tried data.table library with its fread function, but I get an error.

data <- readRDS("myData.rds")

data <- fread("myData.rds")  # error

Answer 1

One way to fasten the read operations of large files is to read it in a compressed mode

system.time(read.table("bigdata.txt", sep=","))

user: 170.901
system: 1.996
elapsed: 192.137

Now trying the same reading but with a compressed file

system.time(read.table("bigdata-compressed.txt.gz", sep=","))

user: 65.511
system: 0.937
elapsed: 66.198

Answer 2

Compression can also influence the speed of reading for rds files:

n<-1000
m<-matrix(runif(n^2), ncol=n)
default<-tempfile()
unComp<-tempfile()
saveRDS(m,default)
saveRDS(m, unComp,compress = F)
microbenchmark::microbenchmark(readRDS(default), readRDS(unComp))
#> Unit: milliseconds
#>              expr      min       lq     mean   median       uq      max neval
#>  readRDS(default) 46.37050 49.54836 56.03324 56.19446 59.99967 96.16305   100
#>   readRDS(unComp) 11.60771 13.16521 15.54902 14.01063 17.36194 27.35329   100
#>  cld
#>    b
#>   a
file.info(default)$size
#> [1] 5326357
file.info(unComp)$size
#> [1] 8000070
require(qs)
#> Loading required package: qs
#> qs v0.25.1.
qs<-tempfile()
qsave(m, qs)
microbenchmark::microbenchmark(qread(qs), readRDS(unComp))
#> Unit: milliseconds
#>             expr       min       lq     mean   median       uq      max neval
#>        qread(qs) 10.164793 12.26211 15.31887 14.71873 17.25536 27.08779   100
#>  readRDS(unComp)  9.342042 12.59317 15.63974 14.44625 17.93492 35.12563   100
#>  cld
#>    a
#>    a
file.info(qs)$size
#> [1] 4187017

However as seen here it comes at the cost of file size. It might also be that the speed of storage has an influence. On slow storage (eg.network, spinning disks) it might actually be better to use compression as the file is quicker read from disk. It is thus work experimenting. Specific packages might even provide slightly better performance here qs has the same speed but a smaller size combining the good of both worlds. For specific data formats other packages might work better see this overview: https://books.ropensci.org/drake/plans.html#special-data-formats-for-targets