Speeding up block of code in python

Question

I have identified two potential reasons why the following code snippet is performing very poorly, given points is a list of 10000 size-2 lists.

1. "append" to add new values given a key

2. The neighbour map dictionary.

    def calculate_distance(point1, point2): a = (point1[0], point1[1]) b = (point2[0], point2[1]) return distance.euclidean(a, b) def get_eps_neighbours(points, eps): neighbours = {} index = 0 for p in points: for q in points: if(calculate_distance(p, q) <= eps): if index in neighbours: neighbours[index].append(q) else: neighbours[index] = q index = index + 1 return {'neighbours': neighbours} 

Any suggestions on how I can improve the code's efficiency?

Answer 1

This is an example of trivially parallel problem.

My recommendation:

• use numpy
• create 2 points^points matrices (2D-arrays), one for x another for y
• use numpy 's array arithmetic

Example:

In [52]: points = [(1,1), (2,2), (3,3), (4,4)]  # super-simple data

In [54]: Xb = numpy.repeat(numpy.array(points)[:,0], 4).reshape(4, 4)

In [60]: Xb
Out[60]: 
array([[1, 1, 1, 1],
       [2, 2, 2, 2],
       [3, 3, 3, 3],
       [4, 4, 4, 4]])

In [61]: Xa = numpy.tile(numpy.array(points)[:,0], 4).reshape(4, 4)

In [62]: Xa
Out[62]: 
array([[1, 2, 3, 4],
       [1, 2, 3, 4],
       [1, 2, 3, 4],
       [1, 2, 3, 4]])

# Yb = numpy.repeat(numpy.array(points)[:,1], 4).reshape(4, 4)
# Ya = numpy.tile(numpy.array(points)[:,1], 4).reshape(4, 4)

In [65]: D = ((Xa - Xb) ** 2 + (Ya - Yb) ** 2) ** 0.5

In [66]: D
Out[66]: 
array([[ 0.        ,  1.41421356,  2.82842712,  4.24264069],
       [ 1.41421356,  0.        ,  1.41421356,  2.82842712],
       [ 2.82842712,  1.41421356,  0.        ,  1.41421356],
       [ 4.24264069,  2.82842712,  1.41421356,  0.        ]])

In [71]: D < 2
Out[71]: 
array([[ True,  True, False, False],
       [ True,  True,  True, False],
       [False,  True,  True,  True],
       [False, False,  True,  True]], dtype=bool)

# Assuming you want only one copy from each pair (a,b), (b,a)
In [73]: triangle = numpy.tri(4, 4, -1, bool)

In [74]: triangle
Out[74]: 
array([[False, False, False, False],
       [ True, False, False, False],
       [ True,  True, False, False],
       [ True,  True,  True, False]], dtype=bool)

In [76]: neighbours = (D < 2) * triangle  # multiplication for "logical and"
Out[76]: 
array([[False, False, False, False],
       [ True, False, False, False],
       [False,  True, False, False],
       [False, False,  True, False]], dtype=bool)

# Neighbours' x and y coordinates are available so:
In [107]: numpy.compress(neighbours.flatten(), Xa.flatten())
Out[107]: array([1, 2, 3])

# Indices to elements in original `points` list like this:
Indexb = numpy.repeat(numpy.arange(4), 4).reshape(4, 4)
Indexa = numpy.tile(numpy.arange(4), 4).reshape(4, 4)
numpy.transpose([numpy.compress(neighbours.flatten(), Indexa.flatten()),
                 numpy.compress(neighbours.flatten(), Indexb.flatten())])
array([[0, 1],
       [1, 2],
       [2, 3]])

Answer 2

With general idea of your algorithm, I think you may reduce the list of points that undergo euclidean distance test by first removeing (or copying to another list) only elements which 2*abs(px - qx) <= eps (repeat for y), that would be much faster than calculating euclidean for all points. If eps is small that would work.

Answer 3

I don't know if this will speed up your code but the pythonic way of counting loops is like this:

for i, p in enumerate(points):

Also - I'm not sure I understand the logic of searching through the whole dictionary (map) keys every time. This bit of code doesn't look like its doing something useful

neighBourMap[index] = q

This adds a key value pair of key: q, value: q to the dictionary. Have you tried just using a list instead ie

neighBourMap = []

Answer 4

All the other answers are correct, but they won't give you a huge speedup. Using numpy arrays will give you a little bit of speedup, parallelizing will give you a speedup. But the speedup will be not enought, if you have 1 million points and you still use your current algorithms, which does n^2 distance calculations. (1 million)^2 is way to many. If your using numpy or not?

You should switch your algorithm. You should store your points in a kd tree. This way you can concentrate your search to a few neighbor candidates. Instead of iterating over all points q , you can simply iterate over all points q with |qx - px| < eps and |qy - py| < eps |qx - px| < eps and |qy - py| < eps |qx - px| < eps and |qy - py| < eps . If your eps is small and there are only a few neighbors for each point, than this should speed you up quite a lot.

Here's a pdf, that describes the algorithm how to find all points in a certain range: http://www.cse.unr.edu/~bebis/CS302/Handouts/kdtree.pdf

Answer 5

You want all combinations of points with each other. You can use itertools.combinations .

Since we're making only the combinations we need, we don't need to keep looking up the dictionary index to append. We can keep the point and its list of neighbours together.

Using a defaultdict with a list means we don't have to manually create the list the first time we look up a point.

Also, you don't actually want the value of the euclidian distance, you just want to know if it is less than some other value. So comparing the squares will give you the same result.

To use a point as a key to a dictionary it needs to be immutable, so we convert it to a tuple:

def distance_squared(a, b):
    diff = complex(*a) - complex(*b)
    return diff.real ** 2 + diff.imag ** 2

from itertools import combinations
from collections import defaultdict

neighbours = defaultdict(list)
eps_squared = eps ** 2

point_neighbours = ((point, neighbours[tuple(point)]) for point in points)

for (p, p_neighbours), (q, _) in combinations(point_neighbours , r=2):
    if distance_squared(p, q) <= eps_squared:
        p_neighbours.append(q)

Answer 6

For one thing you can replace

index in neighBourMap.keys()):`

with just

index in neighBourMap

which will run faster as a copy of the dictionary's key does not need to be created.

Better yet, use a defaultdict(list) which obviates the need for checking for a key before appending to the list value.