python中具有不同排列的大数据集

Question

I have a combination of 50 letters, and I need distinct permutations of all of them printed into a csv file. Right now I was using more_itertools method of distinct_permutations to create the list. Within the 50 letters, 40 of them are the same, and the rest 10 letters are the same. I used Mathematica to check the possible combos, (50!/(40! * 10!)), and there are more than 10 billions of them, so I wonder if the "distinct_permutation" the most efficient way of doing it? Because i was running this code since this morning and it's still running. Thanks.

Answer 1

Are you aware that data will occupy about terabyte on your hard disk? ;)
(and writing will take about 6 hours for usual HD's)

This problem is equivalent to generation of combinations. You can try itertools combinations method. If it is slow too, consider using bit arithmetics.

With only two types of letters the problem is similar to generation of all 50-bit numbers containing 10 ones. There is fast way to produce these bit patterns. During generation convert every bit pattern to letter combination (there are concise ways to map binary to your alphabet in Python, but I don't know the fastest way).

Short example:

def nextperm(v):
    t = (v | (v - 1)) + 1
    w = t | ((((t & -t) // (v & -v)) >> 1) - 1)
    return w

v = 0b0011
print("{0:b}".format(v))
while (v != 0b1100):
    v = nextperm(v)
    print("{0:b}".format(v))

gives output

11   
101
110
1001
1010
1100

that corresponds to

AABB
ABAB    
ABBA
BAAB
BABA
BBAA

In my experiment generation of 10^8 steps (1/100 of your full range) for initial pattern v = 0b00000000000000000000000000000000000000001111111111 without output took 60 seconds

Edit: one more experiment with partial real output . I am sure that building a string might be performed faster, but don't know the best way in Python. My implementation generates 50-MBytes file in 13 seconds (1/10000 of real size), so full generation will take 1.5 days. Good implemenation of string building (and usage of faster language instead of Python) might give gain up to 10 times.

def nextperm(v):
    t = (v | (v - 1)) + 1
    w = t | ((((t & -t) // (v & -v)) >> 1) - 1)
    return w

def writeout(v):
    outs = ""
    t = v
    for i in range(50):
       outs = alphabet[(t & 1)] + outs
       t = t >> 1
    my_file.write(outs + "\n")

v = 0b00000000000000000000000000000000000000001111111111
alphabet = "AB"
my_file = open("out.txt", "w")
for i in range(1000000):
#while (v != 0b11111111110000000000000000000000000000000000000000):
    writeout(v)
    v = nextperm(v)
writeout(v)
my_file.close()

Also you can try to implement 'next permutation' algorithm on numpy arrays of letters to provide faster output.