加快阵列中所有可能对之间的距离

Question

I have an array of x,y,z coordinates of several (~10^10) points (only 5 shown here)

a= [[ 34.45  14.13   2.17]
    [ 32.38  24.43  23.12]
    [ 33.19   3.28  39.02]
    [ 36.34  27.17  31.61]
    [ 37.81  29.17  29.94]]

I want to make a new array with only those points which are at least some distance d away from all other points in the list. I wrote a code using while loop,

 import numpy as np
 from scipy.spatial import distance 

 d=0.1 #or some distance 
 i=0
 selected_points=[]
 while i < len(a):
          interdist=[]  
          j=i+1
          while j<len(a):
              interdist.append(distance.euclidean(a[i],a[j]))
              j+=1

          if all(dis >= d for dis in interdist):
              np.array(selected_points.append(a[i]))
          i+=1

This works, but it is taking really long to perform this calculation. I read somewhere that while loops are very slow.

I was wondering if anyone has any suggestions on how to speed up this calculation.

EDIT: While my objective of finding the particles which are at least some distance away from all the others stays the same, I just realized that there is a serious flaw in my code, let's say I have 3 particles, my code does the following, for the first iteration of i , it calculates the distances 1->2 , 1->3 , let's say 1->2 is less than the threshold distance d , so the code throws away particle 1 . For the next iteration of i , it only does 2->3 , and let's say it finds that it is greater than d , so it keeps particle 2 , but this is wrong! since 2 should also be discarded with particle 1 . The solution by @svohara is the correct one!

Answer 1

For big data sets and low-dimensional points (such as your 3-dimensional data), sometimes there is a big benefit to using a spatial indexing method. One popular choice for low-dimensional data is the kd tree.

The strategy is to index the data set. Then query the index using the same data set, to return the 2-nearest neighbors for each point. The first nearest neighbor is always the point itself (with dist=0), so we really want to know how far away the next closest point is (2nd nearest neighbor). For those points where the 2-NN is > threshold, you have the result.

from scipy.spatial import cKDTree as KDTree
import numpy as np

#a is the big data as numpy array N rows by 3 cols
a = np.random.randn(10**8, 3).astype('float32')

# This will create the index, prepare to wait...
# NOTE: took 7 minutes on my mac laptop with 10^8 rand 3-d numbers
#  there are some parameters that could be tweaked for faster indexing,
#  and there are implementations (not in scipy) that can construct
#  the kd-tree using parallel computing strategies (GPUs, e.g.)
k = KDTree(a)

#ask for the 2-nearest neighbors by querying the index with the
# same points
(dists, idxs) = k.query(a, 2)
# (dists, idxs) = k.query(a, 2, n_jobs=4)  # to use more CPUs on query...

#Note: 9 minutes for query on my laptop, 2 minutes with n_jobs=6
# So less than 10 minutes total for 10^8 points.

# If the second NN is > thresh distance, then there is no other point
# in the data set closer.
thresh_d = 0.1   #some threshold, equiv to 'd' in O.P.'s code
d_slice = dists[:, 1]  #distances to second NN for each point
res = np.flatnonzero( d_slice >= thresh_d )

Answer 2

Here's a vectorized approach using distance.pdist -

# Store number of pts (number of rows in a)
m = a.shape[0]

# Get the first of pairwise indices formed with the pairs of rows from a
# Simpler version, but a bit slow : idx1,_ = np.triu_indices(m,1)
shifts_arr = np.zeros(m*(m-1)/2,dtype=int)
shifts_arr[np.arange(m-1,1,-1).cumsum()] = 1
idx1 = shifts_arr.cumsum()

# Get the IDs of pairs of rows that are more than "d" apart and thus select 
# the rest of the rows using a boolean mask created with np.in1d for the 
# entire range of number of rows in a. Index into a to get the selected points.
selected_pts = a[~np.in1d(np.arange(m),idx1[distance.pdist(a) < d])] 

For a huge dataset like 10e10 , we might have to perform the operations in chunks based on the system memory available.

Answer 3

1. Drop the append, it must be really slow. You can have a static vector of distances and use [] to put the number in the right position.

2. Use min instead of all. You only need to check if the minimum distance is bigger than x.

3. Actually, you can break on your append in the moment that you find a distance smaller than your limit, and then you can drop out both points. In this way you even do not have to save any distance (unless you need them later).

   4. Since d(a,b)=d(b,a) you can do the internal loop only for the following points, forget about the distances you already calculated. If you need them you can pick the faster from the array.

From your comment, I believe this would do, if you have no repeated points.

selected_points = []
for p1 in a:
    save_point = True
    for p2 in a:
        if p1!=p2 and distance.euclidean(p1,p2)<d:
            save_point = False
            break
    if save_point:
        selected_points.append(p1)

return selected_points

In the end I check a,b and b,a because you should not modify a list while processing it, but you can be smarter using some aditional variables.

Answer 4

your algorithm is quadratic (10^20 operations), Here is a linear approach if distribution is nearly random. Splits your space in boxes of size d/sqrt(3)^3 . Put each points in its box.

Then for each box,

• if there is just one point, you just have to calculate distance with points in a little neighborhood.

• else there is nothing to do.