获取字符串中不同索引数的快速方法

Question

I wanted to get number of indexes in two string which are not same.

Things that are fixed:

String data will only have 0 or 1 on any index. ie strings are binary representation of a number.

Both the string will be of same length.

For the above problem I wrote the below function in python

def foo(a,b):
    result = 0
    for x,y in zip(a,b):
        if x != y:
            result += 1
    return result

But the thing is these strings are huge. Very large. So the above functions is taking too much time. any thing i should do to make it super fast.

This is how i did same in c++, Its quite fast now, but still can't understand how to do packing in short integers and all that said by @Yves Daoust :

size_t diff(long long int n1, long long int n2)
{
long long int c = n1 ^ n2;
bitset<sizeof(int) * CHAR_BIT> bits(c);
string s = bits.to_string();

return std::count(s.begin(), s.end(), '1');

}

Answer 1

I'll walk through the options here, but basically you are calculating the hamming distance between two numbers. There are dedicated libraries that can make this really, really fast, but lets focus on the pure Python options first.

Your approach, zipping

zip() produces one big list first , then lets you loop. You could use itertools.izip() instead, and make it a generator expression:

from itertools import izip

def foo(a, b):
    return sum(x != y for x, y in izip(a, b))

This produces only one pair at a time, avoiding having to create a large list of tuples first.

The Python boolean type is a subclass of int , where True == 1 and False == 0 , letting you sum them:

>>> True + True
2

Using integers instead

However, you probably want to rethink your input data. It's much more efficient to use integers to represent your binary data; integers can be operated on directly. Doing the conversion inline, then counting the number of 1s on the XOR result is:

def foo(a, b):
    return format(int(a, 2) ^ int(b, 2), 'b').count('1')

but not having to convert a and b to integers in the first place would be much more efficient.

Time comparisons:

>>> from itertools import izip
>>> import timeit
>>> s1 = "0100010010"
>>> s2 = "0011100010"
>>> def foo_zipped(a, b): return sum(x != y for x, y in izip(a, b))
... 
>>> def foo_xor(a, b): return format(int(a, 2) ^ int(b, 2), 'b').count('1')
... 
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_zipped as f')
1.7872788906097412
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_xor as f')
1.3399651050567627
>>> s1 = s1 * 1000
>>> s2 = s2 * 1000
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_zipped as f', number=1000)
1.0649528503417969
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_xor as f', number=1000)
0.0779869556427002

The XOR approach is faster by orders of magnitude if the inputs get larger, and this is with converting the inputs to int first.

Dedicated libraries for bitcounting

The bit counting ( format(integer, 'b').count(1) ) is pretty fast, but can be made faster still if you installed the gmpy extension library (a Python wrapper around the GMP library ) and used the gmpy.popcount() function:

def foo(a, b):
    return gmpy.popcount(int(a, 2) ^ int(b, 2))

gmpy.popcount() is about 20 times faster on my machine than the str.count() method. Again, not having to convert a and b to integers to begin with would remove another bottleneck, but even then there per-call performance is almost doubled:

>>> import gmpy
>>> def foo_xor_gmpy(a, b): return gmpy.popcount(int(a, 2) ^ int(b, 2))
... 
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_xor as f', number=10000)
0.7225301265716553
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_xor_gmpy as f', number=10000)
0.47731995582580566

To illustrate the difference when a and b are integers to begin with:

>>> si1, si2 = int(s1, 2), int(s2, 2)
>>> def foo_xor_int(a, b): return format(a ^ b, 'b').count('1')
... 
>>> def foo_xor_gmpy_int(a, b): return gmpy.popcount(a ^ b)
... 
>>> timeit.timeit('f(si1, si2)', 'from __main__ import si1, si2, foo_xor_int as f', number=100000)
3.0529568195343018
>>> timeit.timeit('f(si1, si2)', 'from __main__ import si1, si2, foo_xor_gmpy_int as f', number=100000)
0.15820622444152832

Dedicated libraries for hamming distances

The gmpy library actually includes a gmpy.hamdist() function, which calculates this exact number (the number of 1 bits in the XOR result of the integers) directly :

def foo_gmpy_hamdist(a, b):
    return gmpy.hamdist(int(a, 2), int(b, 2))

which'll blow your socks off entirely if you used integers to begin with:

def foo_gmpy_hamdist_int(a, b):
    return gmpy.hamdist(a, b)

Comparisons:

>>> def foo_gmpy_hamdist(a, b):
...     return gmpy.hamdist(int(a, 2), int(b, 2))
... 
>>> def foo_gmpy_hamdist_int(a, b):
...     return gmpy.hamdist(a, b)
... 
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_xor as f', number=100000)
7.479684114456177
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_gmpy_hamdist as f', number=100000)
4.340585947036743
>>> timeit.timeit('f(si1, si2)', 'from __main__ import si1, si2, foo_gmpy_hamdist_int as f', number=100000)
0.22896099090576172

That's 100.000 times the hamming distance between two 3k+ digit numbers.

Another package that can calculate the distance is Distance , which supports calculating the hamming distance between strings directly.

Make sure you use the --with-c switch to have it compile the C optimisations; when installing with pip use bin/pip install Distance --install-option --with-c for example.

Benchmarking this against the XOR-with-bitcount approach again:

>>> import distance
>>> def foo_distance_hamming(a, b):
...     return distance.hamming(a, b)
... 
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_xor as f', number=100000)
7.229060173034668
>>> timeit.timeit('f(s1, s2)', 'from __main__ import s1, s2, foo_distance_hamming as f', number=100000)
0.7701470851898193

It uses the naive approach; zip over both input strings and count the number of differences, but since it does this in C it is still plenty faster, about 10 times as fast. The gmpy.hamdist() function still beats it when you use integers, however.

Answer 2

未经测试，但是如何执行：

sum(x!=y for x,y in zip(a,b))

Answer 3

If the strings represent binary numbers, you can convert to integers and use bitwise operators:

def foo(s1, s2):
    # return sum(map(int, format(int(a, 2) ^ int(b, 2), 'b'))) # one-liner
    a = int(s1, 2) # convert string to integer 
    b = int(s2, 2)
    c = a ^ b # use xor to get differences
    s = format(c, 'b') # convert back to string of zeroes and ones
    return sum(map(int, s)) # sum all ones (count of differences)

s1 = "0100010010"
s2 = "0011100010"
     # 12345

assert foo(s1, s2) == 5

Answer 4

Pack your strings as short integers (16 bits). After xoring, pass to a precomputed lookup table of 65536 entries that gives the number of 1s per short.

If pre-packing is not an option, switch to C++ with inline AVX2 intrinsics. They will allow you to load 32 characters in a single instruction, perform the comparisons, then pack the 32 results to 32 bits (if I am right).