Exception Java heap space

Question

This is my code:

public static void runSGD(double[] R, double[][] theta, double convergenceTol)
{
    List<Integer> allEdges = new ArrayList<Integer>(2*E);    
    for (int i = 0; i < 2*E; i++) 
        allEdges.add(i);
    Collections.shuffle(allEdges, new Random(shuffleSeed));
    double oldRes = calcObj(R, theta, allEdges), newRes = 0.0;
    long numEdges = 0;
    for (int _e = 0; _e < 2*E*tp; _e++) {
        int e = allEdges.get(_e);
        numEdges += weights.get(e);
    }
    if (verbose)
        System.out.printf("[Info] Number of edges in training, including multiplicity = %d\n", numEdges);
    int[][] edgeTable = new int[4][1<<30];
    long part = 0; int cur = 0;
    for (long i = 0; i < numEdges; i++) {
        if (i+1 > part) {
            part += weights.get(allEdges.get(cur));
            cur++;
        }
        int row = (int) (i >>> 30);
        int col = (int) (i & ((1 << 30) -1));
        edgeTable[row][col] = allEdges.get(cur-1);
    }
}

The error is Exception in thread "main" java.lang.OutOfMemoryError: Java heap space when run this code:

 int[][] edgeTable = new int[4][1<<30];

I have try -Xmx1g,-Xmx3g, but didn't work ,how to fix it?

Answer 1

You are allocating 4 int[] arrays of 2^30 ints. That is 2^34 bytes or 16 gigabytes. Clearly that won't fit into a 1 or 3 gigabyte heap. Indeed, a typical laptop or PC won't have enough RAM for this ...

There is a secondary problem about whether the heap spaces are large enough to hold a 2^32 byte object, but it should be possible to address that if you can make the heap large enough.

In fact the JVM supports arrays of just under 2^31 elements; see Do Java arrays have a maximum size? , so the array size per se is not the problem here.

Answer 2

Arrays use ints to address single array elements. The maximum int-value is 2^31-1. You create 4 Arrays of the size 2^30 which means you have 2^32 elements in your array. Java does simple not support arrays of that size.

You can fix it by making 4 distinct arrays.

Also allocating 3GB with -Xmx3g wont help, since your array alone will need 16GB of RAM.

Answer 3

As others mentioned, you are trying to allocate huge arrays. You can try by allocating chunks of small array or try collections.