创建张量的欧几里得距离矩阵

Question

I have 10000 of matrixes with the shape (32, 32, 3). I want to create an euclidean distance matrix between all the matrixes. At the end, it is going to be like,

[0, d2, d3, d4, ...]
[d1, 0, d3, d4, ...]
[d1, d2, 0, d4, ...]
[d1, d2, d3, 0, ...]

How I can make it in the fastest way? I have tried the following, but it takes ages to finish.

import numpy as np
dists = []
for a in range(len(X_test)):
    dists.append([])
    for b in range(len(X_test)):
        dists[a].append(np.linalg.norm(X_test[a] - X_test[b]))
print dists

Answer 1

You can cut the time in half by exploiting the fact that the distance matrix is symmetrical and only compute the upper triangular portion by using using

for b in range(a+1, len(X_test)):

on line 5.

I don't see any other obvious optimizations while keeping the problem exactly the same, but it also seems that you're working with 32x32 images in a three channel format. That's 3072 dimensions! Why not first down-sample to 4x4, convert to HSL color space, and keep only Hue and Lightness to get a (4,4,2) "signature" for each image. If your problem is mostly about shape, you can throw away Hue too and basically work with black-and-white images.

(4,4,2) has only 32 dimensions, for a savings of 100 compared to (32,32,3). And if you did want to do the full comparison in the (32,32,3) space, you could do that only on images that are already very similar in the (4,4,2) space.

Answer 2

I have read Divakar comment .

Rather than asking "Show me Divakar" I asked myself "What is this pdist/cdist stuff?" — I read about pdist and norm and I came out with the following code

Import stuff:

In [1]: import numpy as np
In [2]: from scipy.spatial.distance import pdist

Generate a random sample, not necessarily as large as the OP's one, and reshape it as suggested by Divakar

In [3]: a = np.random.random((100,32,32,3))
In [4]: b = a.reshape((100,32*32*3))

Using the magic of IPython, let's benchmark the two approaches

In [5]: %%timeit
   ...: dists = []
   ...: for i in range(len(a)):
   ...:     dists.append([])
   ...:     for j in range(len(a)):
   ...:         dists[i].append(np.linalg.norm(a[i] - a[j]))
128 ms ± 337 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [6]: %timeit pdist(b)
12.3 ms ± 252 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Divakar's was 1 order of magnitude faster — but what about the accuracy? Let's repeat the computations...

In [7]: dists1 = []
   ...: for i in range(len(a)):
   ...:     dists1.append([])
   ...:     for j in range(len(a)):
   ...:         dists1[i].append(np.linalg.norm(a[i] - a[j]))
In [8]: dists2 = pdist(b)

To compare the results, we must be aware that pdist computes only the upper triangle of the square matrix of distances (because the matrix is symmetric and the principal diagonal is identically equal to zero) so we must be careful in checking our results: hence I check the off diagonal part of the first row of dists1 with the first 99 elements of dists2 using allclose

In [9]: np.allclose(dists1[0][1:], dists2[:99])
Out[9]: True

The result is the same, nice.

What about an estimate of the time required for 10,000 elements? The feeling is that's quadratic, but let's experiment doubling the number of elements

In [10]: b = np.random.random((200,32*32*3))
In [11]: %timeit pdist(b)
48 ms ± 97.7 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [12]: 

the new timing is 4 times the initial one, so my estimate for your computation, on my feeble pc and using Divakar's proposal, is 12ms x 100 x 100 = 120,000ms = 120s. You should read carefully the excellent answer by olooney and decide what you really want to do.