如何更改并行进程数？

Question

I have a python script which runs a method in parallel.

parsers = {
    'parser1': parser1.process,
    'parser2': parser2.process
}

def process((key, value)):
    parsers[key](value)

pool = Pool(4)
pool.map(process_items, items)

process_items is my method and items is a list of tuples with two elements to each tuple. The items list has around 100k items.

process_items will then call a method depending on what parameters are given. My problem being maybe 70% of the list I can run with high parallelism but the other 30% can only run with 1/2 threads otherwise will cause a failure outside of my control.

So in my code I have around 10 different parser processes. For say 1-8 I want to run with Pool(4) but 9-10 Pool(2).

What is the best way to optimise this?

Answer 1

I think your best option is to use two pools here:

from multiprocessing import Pool
# import parsers here

parsers = {
    'parser1': parser1.process,
    'parser2': parser2.process,
    'parser3': parser3.process,
    'parser4': parser4.process,
    'parser5': parser5.process,
    'parser6': parser6.process,
    'parser7': parser7.process,
}

# Sets that define which items can use high parallelism,
# and which must use low
high_par = {"parser1", "parser3", "parser4", "parser6", "parser7"}
low_par = {"parser2", "parser5"}

def process_items(key, value):
    parsers[key](value)

def run_pool(func, items, num_items, check_set):
    pool = Pool(num_items)
    out = pool.map(func, (item for item in items if item[0] in check_set))
    pool.close()
    pool.join()
    return out

if __name__ == "__main__":
    items = [('parser2', x), ...] # Your list of tuples
    # Process with high parallelism
    high_results = run_pool(process_items, items, 4, high_par)
    # Process with low parallelism
    low_results = run_pool(process_items, items, 2, low_par)

Trying to do this in one Pool is possible, through clever use of synchronization primitives, but I don't think it will end up looking much cleaner than this. It's also could end up running less efficiently, since sometimes your pool will need to wait around for work to finish, so it can process a low parallelism item, even when high parallelism items are available behind it in the queue.

This would get complicated a bit if you needed to get the results from each process_items call in the same order as they fell in the original iterable, meaning the results from each Pool need to get merged, but based on your example I don't think that's a requirement. Let me know if it is, and I'll try to adjust my answer accordingly.

Answer 2

You can specify the number of parallel threads in the constructor for multiprocessing.Pool :

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    pool = Pool(5)   # 5 is the number of parallel threads
    print pool.map(f, [1, 2, 3])