Java compressed collection library

Question

Is there a Java third party library that provides a collection that compresses its constituent objects? I googled for it but came up blank. Wouldn't such a construct be useful for large (read multi-gig) maps, or some such thing? Sure there's a performance penalty with access and storage, but for long lived infrequently accessed references, it seems reasonable, no?

Answer 1

The use cases for this are essentially use cases for in-memory databases, so you should probably look into those.

If you were to do this, you'd probably have to basically serialize arbitrary java objects into bytes, then Reflect them back into the classes. May as well use an in-memory database - there's no real difference that I can see anyway, other than that java is probably a bit high-level for that kind of thing.

Note that this is actually somewhat java-specific - it might be that in C you could have a library which would grab the memory and just compress it without having to do any fancy stuff, but since Java doesn't have access to memory it makes that kind of thing a bit hard...

Answer 2

MapDB apparently implements Java collections style maps and can perform "Transparent compression" (See http://www.mapdb.org/apidocs/org/mapdb/DBMaker.html#compressionEnable() ).

I think it's designed for either on disk storage or off-heap storage (See http://www.mapdb.org/apidocs/org/mapdb/DBMaker.html#newDirectMemoryDB() ), so you can choose between a block of disk or a block of non-garbage collected memory.

Answer 3

Unlikely - such a collection wouldn't really be very useful in most situations

Data structures are generally designed to give high performance for a set of specific usage patterns. Adding compression would just add overhead and slow them down for their main use cases. In particular note that most efficient compression algorithms use back-references to previously seen data. This is usually incompatible (ie impossible to implement efficiently) with random access patterns expected from collection classes, and also incompatible with the ability to mutate parts of the collection.

Of course, compression is great for sequential access to large chunks of data, and handling very large data volumes that we need to shuffle between slower storage and main memory. But we already have great tools for that called filesystems and databases :-)