从多进程函数调用中检索和合并返回的数组的有效方法

Question

def get_version_list(env):
  list = []
  #do some intensive work
  return list

if __name__ == '__main__':
  from multiprocessing import Pool

  pool = Pool()

  result1 = pool.apply_async(get_version_list, ['prod'])
  result2 = pool.apply_async(get_version_list, ['uat'])
  #etc, I have six environment to check.

  alist = result1.get()
  blist = result2.get()

Ideally, I would like to have a list containing my six environments, and loop on that list to call my function for each environment (in parallel) then unite all the returned lists. The united list should not contain a value multiple times.

something like so (I knwo that code does not work, but it's to give an idea of what wish to do)

if __name__ == '__main__':
  from multiprocessing import Pool

  pool = Pool()

  env = ['uat', 'prod', 'lt', 'lt2', 'cert', 'sin']

  for e in env:
      result = pool.apply_async(get_version_list, [e])

  #Merging the lists of the 6 function calls (unique entries)
  list = result.get()

Is there a simple way to do it?

Answer 1

Use map_async instead of apply_async .

pool = Pool()
env = ['uat', 'prod', 'lt', 'lt2', 'cert', 'sin']
x = pool.map_async(get_version_list, env)

And now x will be a list of the results.