python中的快速64位确定性哈希

Question

I have previously been using a adler32 to produce a 32-bit hash of blocks of text (which i then use as a filename for saving a cache of the processed version of that text). eg

  hashed_file_name = adler32(pragraph.encode())

I am looking to increase the hash size, to reduce the likelihood of collisions [ie two different blocks of text getting the same hashed-code, which given I have around 10-million text bocks, i think would give a collisions in about 0.2% of cases - ie 10 million / 2^32 ]

My question is, what is the fastest hash generator which produces at least a 64-bit hash? Would sha1 be an efficient option [which produces a 160-bit hash]? ie

  hashed_file_name  = hashlib.sha1(pragraph.encode()).hexdigest()

While this is a over-kill for my neads, are other versions/options more efficient in terms of processing time?

Answer 1

MD5 is 128bit, and doesn't add a dependency. It's probably fast enough, although you know your requirements better than I do. Another thought is to apply your 32-bit hash twice, after permuting the data, say by a XOR or rotation.