Java persistent key-value store

Question

I know this may have been asked a zillion times before but I cannot seem to find the golden solution for my exact use case.

I only have one data structure, a map where the key is a string. The objects of the map are maps themselves but this time the values are simple objects/primitives such as string, int, double, etc. So a map of maps. The keys of the innermost map is constant, ie no entries are ever added/removed from the innermost map except when created. So it is kind of like a traditional table, albeit each row may have arbitrary columns.

I need this data structure to be persistent and replicated.

Here are my requirements:

1. Pure Java solution
2. The disk map is only used in case of re-start. Hence there are never any reads from disk and all the writing is only done by one application)
3. Embedded.
4. Performance. It is the UPDATE performance of existing records that is important. UPDATEs will happen potentially 100k times per second (but more likely 20-50k per second). As for INSERTs/DELETEs they do of course happen but probably only a few times per day. Hence I do not worry too much about INSERT/DELETE performance.
5. Replicated. For resilience I need the disk copy of the map to be replicated. The replication from master to slave does not need to be part of the original transaction, ie I can sacrifice some ACIDness for performance.
6. Number of records is expected to be 100k-200k, but not much higher. The size of each record is probably 100-200 KBytes so really not that much data in total. I'm guessing the total size of the data file will be below 100 MBytes and that is probably an estimate on the high side.
7. The total amount of data is not more than it can always fit in memory. (this is why I can guarantee that there will be no disk reads, except during startup)
8. My application is not distributed. At any given point in time there's only one active process that writes to disk.
9. Liberal open source license. (Apache, BSD, LGPL, should be fine)

The application in question never needs to store anything but the above data structure, ie it will not have a future uncovered need for other persistent data structures. Hence it sounds fair to optimize based on this particular data structure.

I've looked at Berkeley DB Java edition but it fails on requirement #6. I've looked at TokyoCabinet/KoyotoCabinet but it fails on requirement #1.

So what would you recommend?

Answer 1

There are several options, but neo4j seems to match what you want. HBase and Cassandra are also options, but more than you probably need.

Answer 2

Have you looked at Redis ? Its an in-memory "database" (key-value store) that, IMHO, meets all your needs.

Answer 3

Have a look at HazelCast . It meets most of your requirements, except that it's distributed.

Answer 4

I think Chronicle Map is a good match for your case

• pure Java, embeddable key-value store
• In-memory, optionally persisted to disk
• Supports 1M writes per second from a single thread
• Has non-blocking, eventually-consistent replication, optimized for high update rates
• LPGLv3 license