大数据集的R内存管理

Question

I am quite new to R and I am currently working on a dataset (size: 2 GB size). I have stored this dataset in a workspace and whenever I load this dataset into R it consumes more than 90% main memory and hence it becomes difficult and highly time consuming to perform operations like filtering, processing and analyzing the data.

I am mainly using dplyr package to filter and form subsets out of the main dataset, as per dynamic user inputs, but it is taking a lot of time to fetch the data. I have also tried using bigmemory package. While it solves the memory consumption issues, it does not allow the dplyr package functions on big.matrix objects.

So can anyone please let me know how can I filter large datasets quickly with optimal memory consumption?

Thanks!

Answer 1

One approach might be to truncate the table to the columns needed while downloading.

Example

sample1 <- read.csv("https://www.sample-videos.com/csv/Sample-Spreadsheet-500000-rows.csv",
                    header=TRUE)

sample2 <- read.csv("https://www.sample-videos.com/csv/Sample-Spreadsheet-500000-rows.csv",
                    header=TRUE)[, c(1, 3, 5)]
> object.size(sample1)
3272064 bytes
> object.size(sample2)
1073240 bytes

To know which to choose, consult the colnames using

var.names <- names(read.csv("https://www.sample-videos.com/csv/Sample-Spreadsheet-500000-rows.csv",
                    header=TRUE))
> var.names
[1] "Eldon.Base.for.stackable.storage.shelf..platinum"
[2] "Muhammed.MacIntyre"                              
[3] "X3"                                              
[4] "X.213.25"                                        
[5] "X38.94"                                          
[6] "X35"                                             
[7] "Nunavut"                                         
[8] "Storage...Organization"                          
[9] "X0.8" 

Answer 2

I have used data.table and DT packages. Using data.table I created a data.table object which is fast to access and analyse. And using DT package, I used the renderDataTable function to get the details quickly on Shiny dashboard.

Answer 3

I have used data.table and DT packages. Using data.table I created a data.table object that can be accessed and analysed quickly. And using renderDataTable function from DT package, I was able to render the table on Shiny dashboard quickly. Thanks all for your help!!