使用python在Redis中进行键压缩

Question

I have very large data set and I'm looking into using Redis . My data set consists of: sha1 hash and additional n value(s) that is associate with that hash.

I use my sha1 hash as a key inside of Redis and my goal is to compress it somehow). I tried to use zlib and then base64 , but new hash is even longer then original sha1 hash:

[alexus@wcmisdlin02 ~]$ python
Python 2.7.5 (default, Nov 20 2015, 02:00:19) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.sha1('test').hexdigest()
'a94a8fe5ccb19ba61c4c0873d391e987982fbbd3'
>>> len(hashlib.sha1('test').hexdigest())
40
>>> import zlib
>>> len(zlib.compress(hashlib.sha1('test').hexdigest()))
46
>>> import base64
>>> base64.b64encode(zlib.compress(hashlib.sha1('test').hexdigest()))
'eJwFwQkBACAIA8BKIihbHB7pH8G7oAXmnaoUZlwpqwXXVsojnNiT2foB7msLYg=='
>>> len(base64.b64encode(zlib.compress(hashlib.sha1('test').hexdigest())))
64
>>> 

any ideas how to go about it?

---

I'm looking into following as well:

• Redis memory-optimization
• How we cut down memory usage by 82% - Octivi Labs
• Storing hundreds of millions of simple key-value pairs in Redis - Instagram Engineering

Answer 1

Results of a hash function are more or less random data, so you can't really compress them. Depending on how may objects/hashes you're dealing with and how much you care about collisions, you could just use a part of the sha1 value if you want to save space or use an algorithm with a shorter digest size (md5, half of sha2-256). Also, you could not convert the digest from hex and use raw binary and save 50%. Finally, define "very large", it's very likely that the size of the key won't make a big difference.