Calculating inverse of a very large matrix

Question

I am trying to calculate inverse of a very large matrix (11300x21500) in C++. So far I have tried Eigen and Armadillo libraries but both failed at initialization stage, saying that there is not enough memory. Can there be any way to overcome this situation?

Thanks in advance

PS
I should correct the size of the matrix to 21500x21500. As UmNyobe suggested, this is not a square matrix. It is actually the observation matrix, X , and I am trying to calculate ( X ^T X ) ^-1

I have a 8GB memory(in a 64bit system), but I don't think I am making use of all of this memory space. The task manager shows that the memory usage at the time of error is 1GB. Maybe there is a OS command in Windows7 that closes an application when its memory usage exceeds 1GB.

By the way, my original purpose is to run a regression over this observation matrix.

One more thing: most columns in each row of the observation matrix X are zero. Can there be a way to take advantage of this, to limit the memory usage in the inverting operation?

Answer 1

Supposing the matrix is square, what you're probably looking for is an in-place matrix inversion algorithm.

You should check out this .

Answer 2

You cannot inverse a non-square matrix.

http://en.wikipedia.org/wiki/Invertible_matrix

Answer 3

Assuming a (11300 x 11300) Matrix of integer (32 bits), you have

4*(11300^2)/(1024^3) = 0.4757 GB

If you are using double precision then double this number.

If the library is using the Strassen algorithm, which requires additional memory of the same magnitude, then you double the previous number.

So inverting a double-based matrix of this size with Strassen or gaussian will cost you 1.9 GB.

Answer 4

I'd like to propose another solution, which only works if you are not interested in the inverse of the matrix itself but in the product of the inverse with a vector. For example, assume that you would like to find the product of your inverse times a vector v , ie w := (X^TX)^{-1} v . In this case, you are actually looking for a solution to the problem

Find w such that (X^T X) w = v

Using iterative algorithms, it is possible to find w given X and v in the equation above without inverting X . One possibility that comes to my mind is using the Method of Conjugate Gradients . This algorithm can be implemented in roughly 10 lines and only requires to be able to compute the product (X^TX) y with a given vector y . In our case, this can even be done in two steps, ie compute z := X y and in a second step X^T z , which will save space as you do not need to store the product X^TX .

Answer 5

Although you're compiling your program on a 64-bit machine, you should also make sure that you are using the correct 64-bit libraries. Otherwise, the program may be compiled in 32-bit and you'll still get the same memory problems.

As for the computation of the inverse, the inverse function of OpenCV may help. Make sure to use the DECOMP_SVD inverse, as I found it to be more effective with near singular matrices.