Implementing java.util.Map for big data

Question

I'm trying to implement a database backed java.util.Map , most of the interface like put and get was easily implemented however I am having trouble figuring out the best way to implement:

    @Override
    public Set<K> keySet() {
          // TODO Auto-generated method stub
           return null;
    }

    @Override
    public Collection<V> values() {
         // TODO Auto-generated method stub
         return null;
    }

    @Override
         public Set<Map.Entry<K, V>> entrySet() {
        // TODO Auto-generated method stub
        return null;
    } 

My concern would be that keys and values could count to millions records. So I don't think its memory and cpu efficient to fetch and store all "keys" or "values" when these methods are accessed.

What are the options to implement a memory efficient way to implementing these?

What is the strategy to implement an iterator for the entrySet?

Answer 1

老实说，它看起来似乎是一种最好的方法，您还必须以一种有效的方法来检索这些值，而不必尝试将整个数据库拉到内存中并返回一个实现的Set或Collection接口的实例。

Answer 2

I recommend using Oracle's BerkeleyDB Java Edition. The com.sleepycat.collections.StoredContainer.StoredMap class implements the java.util.Map interface and will also backup data to disk. I have used it to work with maps with about 8GB data.

StoredMap: http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/collections/StoredMap.html

BerkeleyDB Java Edition: http://www.oracle.com/technetwork/database/berkeleydb/overview/index-093405.html

Answer 3

If the total data volume is big (Gigabytes), it might be worthwhile to move the data off-heap, to avoid long GC pauses. As a real-world example, see this post: Going off-heap to improve latency and reduce AWS bill .