计算大型列表的方法

Question

Suppose, I need to count collisions for various schemes of hashes. Number of elements in input sequence is 1e10^8 and more. Don't know how to count this analytically, so using brute-force.

It's obvious not to keep this list of hashes in RAM. That is the best way to write a code for my needs? Should i dump it in database or something? What libraries are preferred to use?

Thank you!

Answer 1

I'd suggest keeping a set of files, each one named with a prefix of the hashes contained within it (for example, if you use a prefix length of 6, then the file named ffa23b.txt might contain the hashes ffa23b11d4334 , ffa23b712f3 , et cetera). Each time you read a hash, you append it to the file with the name corresponding to the first N characters of the hash.

You can also use bloom filters to quickly rule out a large fraction of the hashes as unique, without having to store all of the hashes in memory. That way, you only have to fall back to searching through a given prefix file if the check against the bloom filter says that you might have actually seen it before - something that will happen rarely.

Answer 2

Short answer : if you have some gigabytes of RAM, use Python dictionaries, it's the easiest way to implement (and probably the faster to run). You can do something like this:

>>> mydict = {}
>>> for i in some_iterator:
        mydict[i] = ''

And then check if the key exists in the mapping:

>>> 0 in mydict
True

>>> 123456789 in mydict
False

Long answer : you can use a persistent key-value store, like GDBM (it looks like Berkeley DB) or another kind of database -- but this approach will be way slower than using just Python dictionaries; on the other hand, with this approach you'll have persistance (if you need).

You can understand GDBM as a dictionary (key-value store) that is persisted in a single file. You can use it as follows:

>>> import gdbm
>>> kv = gdbm.open('my.db', 'cf')

Then the file my.db will be created (see Python GDBM documentation to understand what cf means).

But it has some limitations, as supporting only strings as keys and values:

>>> kv[0] = 0
Traceback (most recent call last)
[...]
TypeError: gdbm mappings have string indices only

>>> kv['0'] = 0
Traceback (most recent call last)
[...]
TypeError: gdbm mappings have string elements only

>>> kv['0'] = '0'

You can populate a gdbm database with all your keys having a dummy value:

>>> for i in some_iterator:
        kv[str(i)] = ''

And then check if the key exists in the mapping:

>>> '0' in kv
True

>>> '123456789' in kv
False