for循环中的多处理

Question

Suppose I have a dictionary where each element is a quadrilateral defined by a tuple of GPS coordinates and also have tuple containing GPS coordinates of origin and destination points for a bunch of trips:(((origin_latitude, origin_longitude), (dest_latitude,dest_longitude)),((...),(...))). Here is an example for two quadrilaterals and two trips:

dictionary={0:((0,0),(0,1),(1,1),(1,0)),1:((3,3),(3,4),(4,4),(4,3))}
trips=(((0.5,0.5),(3.5,3.5)),((-1,-1),(-2,-2)))

I want to classify each trip into an origin quadrilateral number, a destination quadrilateral number, and a combination number between origin and destination (trip reference number).Here is what I am doing:

import matplotlib.path as mplPath

def is_in_zone(quadri,point):

    bbPath = mplPath.Path(quadri)
    return bbPath.contains_point(point)

def get_zone_nbr(dictio,trip):

    start_zone=-1
    end_zone=-1
    trip_ref=-1

    for key,coordinates in dictio.iteritems():

        if is_in_zone(coordinates,trip[0]):
            start_zone=key
        if is_in_zone(coordinates,trip[1]):
            end_zone=key
        if start_zone>-1 and end_zone>-1:
            trip_ref=len(dictio)*start_zone+end_zone
            break
    return (start_zone,end_zone,trip_ref)

if __name__=='__main__':

    dictionary={0:((0,0),(0,1),(1,1),(1,0)),1:((3,3),(3,4),(4,4),(4,3))}
    trips=(((0.5,0.5),(3.5,3.5)),((-1,-1),(-2,-2)))

    for t in trips:
        get_zone_nbr(dictionary,t)

My dictionary will approximately be of length 30, so the function get_zone_nbr will be quite slow. I have millions of trips to process. Do you see any obvious way to optimize get_zone_nbr()? or anything that would make this code run faster (eg multiprocessing but I am not sure how to use it with loops).

Answer 1

A simple first parallelism is to process your trips in parallel.

>>> import matplotlib.path as mplPath
>>> def is_in_zone(quadri,point):
...     bbPath = mplPath.Path(quadri)
...     return bbPath.contains_point(point)
... 
>>> def get_zone_nbr(dictio,trip):
...     start_zone=-1
...     end_zone=-1
...     trip_ref=-1
...     for key,coordinates in dictio.iteritems():
...         if is_in_zone(coordinates,trip[0]):
...             start_zone=key
...         if is_in_zone(coordinates,trip[1]):
...             end_zone=key
...         if start_zone>-1 and end_zone>-1:
...             trip_ref=len(dictio)*start_zone+end_zone
...             break
...     return (start_zone,end_zone,trip_ref)
... 
>>> dictionary={0:((0,0),(0,1),(1,1),(1,0)),1:((3,3),(3,4),(4,4),(4,3))}
>>> trips=(((0.5,0.5),(3.5,3.5)),((-1,-1),(-2,-2)))
>>> 
>>> from pathos.pools import ThreadPool 
>>> pool = ThreadPool()
>>> 
>>> results = pool.map(lambda x: get_zone_nbr(dictionary, x), trips)
>>> results
[(0, 1, 1), (-1, -1, -1)]

I'm using pathos which is a multiprocessing fork that provides better serialization, flexibility, and interactivity. (I'm also the author.)

You can also then apply the same approach to transform the for loop inside your function get_zone_nbr into a map function call. pathos allows you to use a map call with multiple arguments. Since you are working over a dictionary items, and the items will naturally be unordered, you can use the "unordered iterated map" (in pathos that's uimap , but in multiprocessing it's imap_unordered ).

I also suggest you time your code, to see which of the map calls will be faster. There are several different variants of map call, and several different parallel backends. I used a thread pool above, but there's also going parallel across processes and sockets (the latter will be too slow for your case). pathos provides a uniform API for all of the choices, so you can just code it once, and then drop in any of the other pools/maps until you find one that's the fastest for your case.

Get pathos here: https://github.com/uqfoundation