预测R中的内存使用情况

Question

I have downloaded a huge file from the UCI Machine learning Dataset library. (~300mb).

Is there a way to predict the memory required to load the dataset, before loading it into R memory?

Googled a lot, but everywhere all I could find is how to calculate memory with R-profiler and several other packages, but after loading the objects into R.

Answer 1

based on "R programming" coursera course, U can calculate the proximate memory usage using number of rows and columns within the data" U can get that info from the codebox/meta file"

memory required = no. of column * no. of rows * 8 bytes/numeric

so for example if you have 1,500,00 rows and 120 column you will need more than 1.34 GB of spare memory required

U also can apply the same approach on other types of data with attention to number of bytes used to store different data types.

Answer 2

If your data's stored in a csv file, you could first read in a subset of the file and calculate the memory usage in bytes with the object.size function. Then, you could compute the total number of lines in the file with the wc command-line utility and use the line count to scale the memory usage of your subset to get an estimate of the total usage:

top.size <- object.size(read.csv("simulations.csv", nrow=1000))
lines <- as.numeric(gsub("[^0-9]", "", system("wc -l simulations.csv", intern=T)))
size.estimate <- lines / 1000 * top.size

Presumably there's some object overhead, so I would expect size.estimate to be an overestimate of the total memory usage when you load the whole csv file; this effect will be diminished if you use more lines to compute top.size . Of course, this approach could be inaccurate if the first 1000 lines of your file are not representative of the overall file contents.

Answer 3

R has the function object.size(), that provides an estimate of the memory that is being used to store an R object. You can use like this:

  predict_data_size <- function(numeric_size, number_type = "numeric") {
  if(number_type == "integer") {
    byte_per_number = 4
  } else if(number_type == "numeric") {
    byte_per_number = 8 #[ 8 bytes por numero]
  } else {
    stop(sprintf("Unknown number_type: %s", number_type))
  }
  estimate_size_in_bytes = (numeric_size * byte_per_number)
  class(estimate_size_in_bytes) = "object_size"
  print(estimate_size_in_bytes, units = "auto")
}
# Example
# Matrix (rows=2000000, cols=100)
predict_data_size(2000000*100, "numeric") # 1.5 Gb