使用python进行多重处理以找到最大值

Question

I'm working with Python 2.7.5 and OpenCV. I have a test image and I want to find it's most similar image in an array of images. I have written a function using OpenCV that will give me the total number of similarity points. The more similar points I have the more similar the images are. Unfortunately this is a rather time consuming function so I would like to parallelize my code to make it faster.

#img is the image that I am trying to find the most number of similar pointswith
maxSimilarPts = 0;

#testImages is a list of testImages
for testImage in testImages:
    #getNumSimilarPts returns the number of similar points between two images
    similarPts = getNumSimilarPts(img, testImage) 

    if similarPts > maxSimilarPts:
        maxSimilarPts = similarPts

How can I do this in parallel with python? Any help would be greatly appreciated.

Answer 1

The following is a (untested) parallel version of the original code. It runs 5 workers in parallel. Each one takes an image from the input queue, calculates the similary, then puts the value and image onto an output queue. When all the workers are done, there are no more images, then the parent process prints the (similarity, imageID) of the most similar image.

# adapted from Raymond Hettinger
# http://stackoverflow.com/questions/11920490/how-do-i-run-os-walk-in-parallel-in-python/23779787#23779787

from multiprocessing.pool import Pool
from multiprocessing import JoinableQueue as Queue
import os, sys


def parallel_worker():
    while True:
        testImage = imageq.get()
        similarPts = getNumSimilarPts(img, testImage) 
        similarq.put( [similarPts, testImage] )
        imageq.task_done()

similarq = Queue()
imageq = Queue()
for testImage in testImages:
    imageq.put(testImage)

pool = Pool(5)
for i in range(5):
    pool.apply_async(parallel_worker)

imageq.join()
print 'Done'

print max(similarq)

Answer 2

Important note:

This code will work natively only on python3. to run it on python2 you must install the concurrent.futures PyPI package .

from concurrent.futures import ProcessPoolExecutor


def multiprocess_max(iterable, key):
    with ProcessPoolExecutor() as executor:
        return max(executor.map(lambda item: (item, key(item)), iterable),
                   key=lambda item: item[1])[0]

The idea behind is the following:

The expensive process is calculating the key for comparing the item. So, what not to calculate the key by multi processes but comparing it using only one process?

Here's how it works:

Create a concurrent.futures.ProcessPoolExecutor , which is a easy-to-use wrapper around the multiprocessing module, and provide a map() function like the builtin but that works concurrently.

Then, from the collections, create for each item tuple with 2 elements: the original item (what we want to return, if it's key is the max) and the key, computed with the passed key function.

After we got a result, pass it to the builtin max() - but we have a problem: the collections now is a collection of tuples! So, we pass a key function that returns the second item - the computed key.

Finally, since max() returns the whole item (which includes the key that is unwanted), we extract the first item - the original item - from its result and return it.

Edit:

After this code locked in my console (the IDLE; I find this question because I needed it too), I thought my solution is wrong :-)

But I wrong, not the solution. This solution won't work in the interpreter . From the docs :

The __main__ module must be importable by worker subprocesses. This means that ProcessPoolExecutor will not work in the interactive interpreter.