什么是在映射中存储大量字符串的最节省内存的方法？

Question

I want to store huge amounts of Strings in a Map<String, MagicObject> , so that the MagicObjects can be accessed quickly. There are so many entries to this Map that memory is becoming a bottleneck. Assuming the MagicObjects can't be optimized, what is the most efficient type of map I could use for this situation? I am currently using the following:

gnu.trove.map.hash.TCustomHashMap<byte[], MagicObject>

Answer 1

If your keys are long enough and have a lot of long enough common prefixes then you can save memory by using a trie (prefix tree) data structure. Answers to this question point to aa couple of Java implementations of trie.

Answer 2

为了开放您的想法，请考虑使用霍夫曼编码先将您的字符串压缩后再放入地图，只要您的字符串固定（字符串的数量和内容不变）即可。

Answer 3

I'm a little late to this party but this question came up in a related search and piqued my interest. I don't usually answer Java questions.

There are so many entries to this Map that memory is becoming a bottleneck.

I doubt it.

For the storage of strings in memory to become a bottleneck you need an awfully large number of unique strings[1]. To put things into perspective, I recently worked with a 1.8m word dictionary (1.8m unique english words) and they took up around 1.6MB in RAM at runtime.

If you used every word in the dictionary as a key you'll still only use 1.6MB of RAM[2] to store the keys, hence memory cannot be your bottleneck.

What I suspect you are experiencing is the O(n^2) performance of string matching. By this I mean that as more keys are added performance slows down exponentially[3]. This is unavoidable if you are using strings are keys.

If you want to speed things up a bit, store each key into a hashtable that doesn't store duplicates and use the hash key as the key to your map.

NOTES:

[1] I'm assuming the strings are all unique or else you would not attempt to use them as a key into a map.

[2] Even if Java uses 2 bytes per character, it still only comes to 3.2MB of memory, total.

[3] It slows down even more if you choose the wrong data structure, such as an unbalanced binary tree, to store your values. I don't know how map stores values internally, but an unbalanced binary tree will have O(2^n) performance - pretty much the worst performance you can find.