简体   繁体   中英

Efficient way to retrieve and merge returned arrays from multiprocess function call

def get_version_list(env):
  list = []
  #do some intensive work
  return list

if __name__ == '__main__':
  from multiprocessing import Pool

  pool = Pool()

  result1 = pool.apply_async(get_version_list, ['prod'])
  result2 = pool.apply_async(get_version_list, ['uat'])
  #etc, I have six environment to check.

  alist = result1.get()
  blist = result2.get()

Ideally, I would like to have a list containing my six environments, and loop on that list to call my function for each environment (in parallel) then unite all the returned lists. The united list should not contain a value multiple times.

something like so (I knwo that code does not work, but it's to give an idea of what wish to do)

if __name__ == '__main__':
  from multiprocessing import Pool

  pool = Pool()

  env = ['uat', 'prod', 'lt', 'lt2', 'cert', 'sin']

  for e in env:
      result = pool.apply_async(get_version_list, [e])

  #Merging the lists of the 6 function calls (unique entries)
  list = result.get()

Is there a simple way to do it?

Use map_async instead of apply_async .

pool = Pool()
env = ['uat', 'prod', 'lt', 'lt2', 'cert', 'sin']
x = pool.map_async(get_version_list, env)

And now x will be a list of the results.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM