Java memory 优化了 [Key:Long, Value:Long] 存储非常大（500M）的并发读取访问

Question

I have a use-case where I need to store Key - Value pairs of size approx. 500 Million entries in sinle VM of size 8 GB. Key and Value are of type Long. Key is auto incremented starting from 1, 2,3, so on..

Only once I build this Map[KV] structure at the start of program as a exclusive operation, Once this is build, used only for lookup, No update or delete is performed in this structure.

I have tried this with java.util.hashMap but as expected it consumes a lot of memory and program give OOM: Heap usage exceeds Error.

I need some guidance on following which helps in reducing the memory footprint, I am Ok with some degradation in access performance.

1. What are the other alternative (from java collection or other libraries) that can be tried here.
2. What is a recommended way to get the memory footprint by this Map, for comparison purpose.

Answer 1

Just use a long[] or long[][] .

500 million ascending keys is less than 2^31. And if you go over 2^31, use a long[][] where the first dimension is small and the second one is large.

(When the key type is an integer, you only need a complicated "map" data structure if the key space is sparse.)

The space wastage in a 1D array is insignificant. Every Java array node has 12 byte header, and the node size is rounded up to a multiple of 8 bytes. So a 500 million entry long[] will take so close to 500 million x 8 bytes == 4 billion bytes that it doesn't matter.

However, a JVM typically cannot allocate a single object that takes up the entire available heap space. If virtual address space is at a premium, it would be advisable to use a 2-D array; eg new long[4][125_000_000] . This makes the lookups slightly more complicated, but you will most likely reduce the memory footprint by doing this.

---

If you don't know beforehand the number of keys to expect, you could do the same thing with a combination of arrays and ArrayList objects. But an ArrayList has the problem that if you don't set an (accurate) capacity, the memory utilization is liable to be suboptimal. And if you populate an ArrayList by appending to it, the instantaneous memory demand for the append can be as much as 3 times the list's current space usage.

Answer 2

There is no reason for using a Map in your case.

If you just have a start index and further indizes are just constant increments, just use a List :

List<Long> data=new ArrayList<>(510_000_000);//capacity should ideally not be reached, if it is reached, the array behind the ArrayList needs to be reallocated, the allocated memory would be doubled by that

data.add(1337L);//inserting, how often you want

long value=data.get(1-1);//1...your index that starts with 1, -1...because your index starts with 1, you should subtract one from the index.

If you don't even add more elements and know the size from the start, an array will be even better:

long[] data=long[510_000_000];//capacity should surely not be reached, you will need to create a new array and copy all data if it is higher
int currentIndex=0;

data[currentIndex++]=1337L//inserting, as often as it is smaller than the size

long value=data[1-1];//1...your index that starts with 1, -1...because your index starts with 1, you should subtract one from the index.

Note that you should check the index ( currentIndex ) before inserting so that it is smaller than the array length.

When iterating, use currentIndex+1 as length instead of .length .

Create an array with the size you need and whenever you need to access it, use arr[i-1] ( -1 because your indizes start with 1 instead of zero).

If you "just" have 500 million entries, you will not reach the integer limit and a simple array will be fine.

If you need more entries and you have sufficient memories, use an array of arrays.

The memory footprint of using an array this big is the memory footprint of the data and a bit more.

However, if you don't know the size, you should use a higher length/capacity then you may need. If you use an ArrayList , the memory footprint will be doubled (temporarily tripled) whenever the capacity is reached because it needs to allocate a bigger array.

A Map would need an object for each entry and an array of lists for all those object that would highly increase the memory footprint. The increasing of the memory footprint (using HashMap ) is even worse than with ÀrrayList s as the underlaying array is reallocated even if the Map is not completely filled up.

But consider saving it to the HDD/SSD if you need to store that much data. In most cases, this works much better. You can use RandomAccessFile in order to access the data on the HDD/SSD on any point.