如何快速从大量数字中生成所有对的列表？

Question

I create a list with numbers from 0 to 131072:

x = [i for i in range(131072)] 

Then all pairs, except for pairs of the same numbers:

pairs = []
append_pairs = pairs.append
for i in range(len(x)):
    for j in range(len(x)):
        if x[i]!=x[j]:
           x2 = [x[i], x[j]] 
           append_pairs(x2)

which gives:

pairs = [[0, 1], [0, 2], [0, 3], ... [131071, 131070]]

But in this syntax it takes a very very long time. Can it be done faster?

Answer 1

You can use itertools.combinations but that will probably also take a little while like so:

import itertools as it

n = 131072
pairs = it.combinations(range(n), 2)

Note that the code above will not give you the list of all pairs but a generator over pairs:

>>> pairs
<itertools.combinations at 0x7fb939a72a48>

You can get the list using

pairs = list(it.combinations(range(n), 2)

Using numpy is probably faster:

import numpy as np

pairs = np.transpose(np.triu_indices(n, 1))

However, the number of pairs you want to generate is enormous and you cannot store the numbers in memory (unless you have a very powerful machine). In particular, you get n * (n - 1) / 2 pairs. If you store the numbers as 8-byte integers, you're looking at just under 70 GB of memory.

For n = 5000 :

• Itertools: 818 ms ± 15.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
• Numpy: 254 ms ± 30.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
• Original method: 3.72 s ± 72.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Note: Because there is more in-built code available, I have generated distinct pairs.