bigmatrix: Error in CreateFileBackedBigMatrix(as.character(backingfile), as.character(backingpath)

Question

After many hours of reading I found the best way to deal with big (really big matrix) in R is to use bigalgebra

I want to store a large 80,000 x 80,000 matrix of 0s that doesn't fit in RAM and then replace its cells by the result of a large matrix multiplication.

I have this error

Error in CreateFileBackedBigMatrix(as.character(backingfile), as.character(backingpath)
Problem creating filebacked matrix.

After doing

my_matrix <- filebacked.big.matrix(ncol = ncol(Xt), nrow = nrow(X), 
                 type = "double", backingpath = getwd(), 
                 descriptorfile = "X.bin.desc", backingfile = "X.bin")

This is a reproducible similar case of my data

X <- matrix(rexp(200, rate=.1), ncol = 11, nrow = 80000)
Xt <- t(X)

So my full chunk is

library(bigalgebra)

X <- matrix(rexp(200, rate=.1), ncol = 11, nrow = 80000)
Xt <- t(X)

my_matrix <- filebacked.big.matrix(ncol = ncol(Xt), nrow = nrow(X), 
                 type = "double", backingpath = getwd(), 
                 descriptorfile = "X.bin.desc", backingfile = "X.bin")

However if I run this (not useful in my case)

my_matrix <- filebacked.big.matrix(ncol = ncol(X), nrow = nrow(Xt), 
                 type = "double", backingpath = getwd(), 
                 descriptorfile = "X.bin.desc", backingfile = "X.bin")

it runs without warnings and stores a small 11 x 11 matrix.

I'm using OS El Capitan with 12 GB RAM and I'm aware that $X^t * X$ would need ~47 GB of RAM to compute, that's why I'm using bigalgebra to store that in the HDD.

Answer 1

Check your available hard drive space, because the error you are signaling could be due to insufficient hard drive space to write down the backing-file necessary to the filebacked.big.matrix. I had your same error and I solved it by indicating an external hard drive as "backingpath".