python和多处理

Question

I have three functions in the python that each one puts an image (image path) as input and makes a simple image processing and creates a new image (image path) as output.

in the example below, one function depends on the other, ie: the function of alg2 takes as input the image that generates the function of alg and the function of alg3 assign as input the image that generates the function of alg2 which depends on the function of alg1.

(I hope you do not mind basically)

because of their relatively high execution time (image processing is that) I would like to ask if I can to parallelize them using python multiprocessing. I have read about multiprocessing map and pool but I was pretty confused .

whenever I summarize I have three interdependent functions and I would like to run them together if done. I would also like to know how I would perform these three functions in a contemporary way if they were not interdependent, ie each was autonomous.

def alg1(input_path_image,output_path_image):
    start = timeit.default_timer()
    ###processing###)
    stop = timeit.default_timer()
    print stop - start
    return output_path_image

def alg1(output_path_image,output_path_image1):
    start = timeit.default_timer()
    ###processing###
    stop = timeit.default_timer()
    print stop - start
    return output_path_image1

def alg3(output_path_image1,output_path_image2):
    start = timeit.default_timer()
    ###processing###
    stop = timeit.default_timer()
    print stop - start
    return output_path_image2

if __name__ == '__main__':
   alg1(input_path_image,output_path_image)
   alg2(output_path_image,output_path_image1)
   alg3(output_path_image1,output_path_image2)

Answer 1

Here is what I would do:

I would split the list of images into smaller parts. Then I would make one function out of those three functions (by making the other 2 functions as private - just for the sake of simplicity). Then you can speed up the whole process by doing:

from multiprocessing import Process

image_list = this_is_your_huge_image_list
# create smaller image lists e.g. [[1, 2, 3], [4, 5, 6], ..]
chunked_lists = [image_list[x:x+100] for x in xrange(0, len(image_list), 100)]

for img_list in chunked_lists:
    p = Process(target=your_main_func, args=(img_list,))
    p.start()
    # without .join() here

Answer 2

It sounds like you're doing something CPU intensive, so you'll need to use the multiprocessing.Process object, rather than threading.Thread . Because of this, you can't return from multiprocessing.Process , and therefore will need to use a multiprocessing.Manager .

So this is an adaptation of your code which will work with multiprocessing.Process :

from multiprocessing import Process, Manager

def alg1(input_path_image,output_path_image, return_dict):
    start = timeit.default_timer()
    ###processing###)
    stop = timeit.default_timer()
    print stop - start
    return_dict['algo1'] = output_path_image

def alg2(output_path_image,output_path_image1, return_dict):
    start = timeit.default_timer()
    ###processing###
    stop = timeit.default_timer()
    print stop - start
    return_dict['algo2'] = output_path_image1

def alg3(output_path_image1,output_path_image2, return_dict):
    start = timeit.default_timer()
    ###processing###
    stop = timeit.default_timer()
    print stop - start
    return_dict['algo3'] = output_path_image2

if __name__ == '__main__':
    manager = Manager()
    return_dict = manager.dict()
    a1 = Process(target=alg1, args=(output_path_image,output_path_image, return_dict))
    a2 = Process(target=alg2, args=(output_path_image1,output_path_image1, return_dict))
    a3 = Process(target=alg3, args=(output_path_image2,output_path_image2, return_dict))
    jobs = [a1, a2, a3]
    for job in jobs:
        job.start()
    for job in jobs:
        job.join()
    a1_return = return_dict['algo1']
    a2_return = return_dict['algo2']
    a3_return = return_dict['algo3']

You'll need to modify this further to give your print statements a little more distinction. At the moment, they will only print a number, and you won't be able to distinguish between them.