python多处理慢

Question

I have some code that parallelizes calls to a function. Inside the function, I check if a file exists, if not then I create it, else I do nothing.

I find that if the files do exist, then calling multiprocessing.process has a fairly huge time penalty compared to a simple for loop. Is this expected or is there something I can do to reduce the penalty?

def fn():
    # Check if file exists, if yes then return else make the file
    if(not(os.path.isfile(fl))):
        # processing takes enough time to make the paralleization worth it
    else:
        print 'file exists'


pkg_num = 0
total_runs    = 2500
threads = []

while pkg_num < total_runs or len(threads):
    if(len(threads) < 3 and pkg_num < total_runs):
        t = multiprocessing.Process(target=fn,args=[])
        pkg_num = pkg_num + 1
        t.start()
        threads.append(t)
    else:
        for thread in threads:
            if not thread.is_alive():
                threads.remove(thread)

Answer 1

There's a fair bit of overhead to bringing up processes -- you've got to weigh the overhead of creating those processes against the performance benefits that you'll gain from making the tasks concurrent. I'm not sure that there's enough of a benefit for a simple OS call for it to be worthwhile.

Also, for the sake of future generations, you should really check out concurrent.futures.ProcessPoolExecutor ; way, way cleaner. If you use 2.7, you can back port it.