Java OutOfMemoryError with ArrayList<List<Integer>>

Question

I want to create a very large graph (with ~10 million edges) in Java. I plan to List<List<Integer>> to describe the edges, with the inside List<Integer> describing the two vertices of each edge (and the vertices are Integer type).

The following code throws the OutOfMemoryError after about 1 million edges are added to the graph. (I simplified how the edge is generated for the sake of discussion.)

public static void main(String[] args) {
  List<List<Integer>> graph = new ArrayList<List<Integer>>();
  for (int i = 0; i < 10000000; i++) {
    List<Integer> edge = new ArrayList<Integer>();
    // the real edges are more complicated (than from vertex i to vertex i+1)
    // this is simplified for the sake of the discussion here
    edge.add(i);
    edge.add(i+1);
    graph.add(edge);
  }
}

I have searched for OutOfMemoryError , and I have increased the initial heap size to 2G for Eclipse: -Xms2g -Xmx4g -Xss2m (which get passed to JVM). But that did not solve the problem.

Then I thought maybe I should garbage collect the List<Integer> edge variable, by calling System.gc() , in case its memory does not get cleared. That did not work either.

I was thinking maybe the problem is with the List<List<Integer>> data structure. I tried List<int[]> , which lasted a bit longer: more edges are added before OutOfMemoryError happens. I do not have a better idea right now.

I have searched around for similar problems, but have not find much help. I wonder if anyone has experience with this kind of situation.

Answer 1

To let your program use more memory from Eclipse:

Go to Run -> Run Configurations. You will see this window

Click on Arguments

Enter your arguments to the VM

Answer 2

Since you are using a lot of RAM besides setting the max heap parameter, make sure you use 64-bit Java. The 32-bit is limited to 2 Gigs or something like that.

Also, for large graphs you should consider using a database.

And last but not least, maybe you can rethink your algorithm, sometimes you just don't need all the nodes and edges.