Python2在简单的数学运算中表现不佳

Question

I am trying to do a small variation of euclidean distance (I am fully aware that that is not euclidean formula) for color quantization from RGB values to a 16 color palette. I have no problems with code, but performance in python is over 25 times slower than Java.

Main method in python looks like this:

def getBaseColor(rValue=128, gValue=128, bValue=128):
 allDistances=[450]*16
 for x in range(0,16):
  valoresColor = coloresWeb[x]
  allDistances[x]= (abs(valoresColor[0]-rValue) + abs(valoresColor[1]-gValue) + abs(valoresColor[2]-bValue))
 return allDistances.index(min(allDistances))

I did small benchmark tests (1M operations) and Java is 25 times faster than Python (2.7.9). Using pypy helps a lot, but still very far from Java.

Python 2: ~5.2s

Java: ~0.2s

pypy: ~ 0.6s

My question is: Am I doing something wrong with python, or it is just super slow by nature?. This is a process that needs to be run hundred of million of times, and no, this is not image processing (although it looks like).

Fully functional minimal codes are provided here for Python and Java.

Answer 1

With NumPy, calculating all million points at the same time:

import time
import numpy as np

webColours = np.array([
    [0,0,0],
    [0,0,128],
    [0,128,0],
    [0,128,128],
    [128,0,0],
    [128,0,128],
    [128,128,0],
    [192,192,192],
    [128,128,128],
    [0,0,255],
    [0,255,0],
    [0,255,255],
    [255,0,0],
    [255,0,255],
    [255,255,0],
    [255,255,255]
])

def getBaseColours(colours):
    # colours is 1000000x3
    # set up a distances array (16x1000000)
    distances = np.zeros((16, np.size(colours, 0)))
    for colour in xrange(16):
        # calculate distance of each input colour to this webColour
        distances[colour] = np.sum(abs(colours - webColours[colour]), 1)
    # which of 16 distances is the least for each of 1000000 colours
    return np.argmin(distances, 0)

startTime = time.time()
colour = np.array([134,234,43])
colours = np.tile(colour, (1000000, 1))
getBaseColours(colours)
print "Time: " + str(time.time()-startTime)

Time: 0.9s on my system (where your Python code executes in 9s). Also, I'm a newbie at NumPy, so possibly the code could be optimised even further.

Answer 2

Since you only want to find a nearest neighbor for color quantization, you don't actually need to to calculate all the distances the way you are doing. In particular, using a KDTree in this case would be much more efficient.

Otherwise, as others have noted you get a slow results for Python because such operations would not normally be performed in pure Python. The default approach would be to use Numpy, and in this case this can also be speed up using a specialized function from Scipy (see scipy.spatial.distance or better in this case scipy.spatial.cKDTree ). Finally if that is still not good enough, you can use Cython, Pypy, etc.

Answer 3

Plain CPython is slow by nature - it comes from the very design of the interpreter. Speaking simplified, CPython is a C++ program that constantly reads your instructions from a file, parses them and acts accordingly.

So for every instruction, you have a full "context switch" from your code down to its representations in C++, including all name lookups, transformations for wrappers, then the actual computation and then back to your code again. Especially loops are costly, because it means you are doing the same again and again. Since CPython is acting per-line, it cannot do any optimization, such as prefetching data, vectorizing etc.

The upside is that you can do powerful introspection and self-modification, with a very simple implementation. The downside is that the interpreter has to go all the way at every step.

In contrast, both Java and PyPy are just-in-time compiled. When they go through a loop, they will realize that they did the same thing already (instruction wise) and be prepared for it. This is why PyPy may be slower than CPython sometimes: it needs a warm-up phase in which it can actually optimize repeated operations. If operations are repeated only a bit or never, there is no advantage.

---

Disclaimer: This is a very simplified view of the CPython interpreter. For example, there are some "short-circuit" instructions such as list comprehensions which are handled more efficiently than regular loops. As these can still call arbitrary code, they are limited in performance as well, however.