out of memory exception caused by big array in java

Question

I'm implenmenting an algorithm which based on probabilistic latent semantic indexing(plsa) and the paper is here and it need a four dimension array which named p_z_d_wt_wv, z is topic, d is document, wt is text word, wv is visual word,and the number of each dimension is about 12, 7000,100, 500, and the array is a double array, so it need 32G memory !! I allocate this memory like this way below, and it is just for demonstration as the number of wt and wv in each document is different.

p_z_d_wt_wv = new double[12][7000][][]; 
for( int t = 0; t < 12; ++t) 
{ 
    for( int d = 0; d < 7000; ++d ) 
    { 
        p_z_d_wt_wv[t][d] = new double[100][500];
    } 
}

when I run the code, it has out of memory problem. First, why do my code run out of memory? Are the memory allocated consecutively if the array are allocated in my way? Is it because java have a memory limit for consecutive memory? If so, what's the limit?

Second, what can I do to solve this problem supposed that the memory of the server is big enough. I know I can change it as a float array, but are there any other solutions?

Answer 1

If you actually need all of that memory, well, you need all of that memory.

There are some alternatives:

1. You could look into using memory mapped files.

2. If the array has a lot of zeros in it, you could store it as a sparse matrix representation (don't explicitly store 0s).

3. If you don't need the whole thing in memory at once, you could also store it in some sort of persistent storage (file, database, etc) and only access the parts you need at any given time.

Answer 2

Are the memory allocated consecutively if the array are allocated in my way? Is it because java have a memory limit for consecutive memory? If so, what's the limit?

No, the JVM can not allocate memory for your array. if you use float for your array, you must set the maximum memory heap space 16GB. You can use file to store your array.