在非常大的数据集上用python生成n选择2个组合

Question

I need to create n choose 2 combinations and am currently using pythons itertools.combinations module.

For a single list of 30,000 strings, creating the combinations runs for hours and uses many gigs of ram, ie

list(itertools.combinations(longlist,2))

is there a method of generating combinations that is potentially better optimized for large objects in memory? Alternately is there a way of using numpy to speed up the process?

Answer 1

you can instantly know how many combinations there are by using binomial coeficient there are (30k choose 2) way to solve this = math.factorial(30000)//(math.factorial(2)*math.factorial(30000-2)) = 449985000 combinations

that said itertools returns a generator so you can iterate over it without loading all the combinations in memory into one big list

Answer 2

I'd use a generator based on np.triu_indices
These are the indices of the upper trianle of an nxn square matrix, where n = len(long_list)

The problem is that the entire set of indices are created first. itertools does not do this and only generates each combination one at a time.

def combinations_of_2(l):
    for i, j in zip(*np.triu_indices(len(l), 1)):
        yield l[i], l[j]

long_list = list('abc')
c = combinations_of_2(long_list)
list(c)

[('a', 'b'), ('a', 'c'), ('b', 'c')]

---

To get them all at once

a = np.array(long_list)
i, j = np.triu_indices(len(a), 1)
np.stack([a[i], a[j]]).T

array([['a', 'b'],
       ['a', 'c'],
       ['b', 'c']], 
      dtype='<U1')

---

timing
long_list = pd.DataFrame(np.random.choice(list(ascii_letters), (3, 1000))).sum().tolist()