How to store 100 million integers in a Java array?

Question

I am working on a small task where I am required to store around 1 billion integers in an Array. However, I am running into a heap space problem. Could you please help me with this?

Machine Details : Core 2 Duo Processor with 4 GB RAM. I have even tried -Xmx 3072m . Is there any work around for this? The same thing works in C++ , so there should definitely be a way to store this many numbers in memory.

Below is the code and the exception I am getting :

public class test {
    private static int C[] = new int[10000*10000];

    public static void main(String[] args) {
        System.out.println(java.lang.Runtime.getRuntime().maxMemory()); 

    }

}

Exception : Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at test.(test.java:3)

Answer 1

Use an associative array. The key is an integer, and the value is the count (the number of times the integer has been added to the list).

This should get you some decent space savings if the distribution is relatively random, much more so if it's not.

Answer 2

If you need to store 1 billion completely random integers then I am afraid that you really do need to corresponding space, ie about 4GB of memory for 32-bit int numbers. You can try increasing the JVM heap space but you need to have a 64-bit OS and at least as much physical memory - and there is only so far that you can go.

On the other hand, you might be able to store those number more efficiently if you can make use of specific constraints within your application.

Eg if you only need to know if a specific int is contained in a set, you could get away with a bit set - ie a single bit for each value in the int range. That is about 4 billion bits, ie 512 MB - a far more reasonable space requirement. For example, a handful of BitSet objects could cover the whole 32-bit integer range without you having to write any bit-handling code...

Answer 3

May be using memory mapped files will help? They are not allocated from the heap. Here is an article how to create a matrix. An array should be easier.

Using a memory mapped file for a huge matrix - Peter Lawrey

Answer 4

You can increase to 4GB on a 32 bit system. If you're on a 64 bit system you can go higher.

Type in cmd this

java -Xmx4g programname

Answer 5

As array so big may not fit into your RAM, you need to configure the sufficient HDD swap space. 4 - 16 Gb on swap do not look like something unrealistic at these times.

Java only allows to use int as an index of array, not long . Hence the largest possible array could have 2147483648 values, enough.

Use -Xmx to raise the memory ceiling that by default will probably be insufficient. 3072m is not enough as one billion ints requires about 4 Gb. As space is needed also for the operating system and the like, that machine with 4 Gb RAM cannot hold all 4 Gb data structure in memory.

JRE or OS may also refuse to grant an piece of memory so big in one go, requiring to allocate in some smaller chunks (maybe array or arrays).