Python多处理共享列表

Question

I'm trying to create a script which parses a file and converts it into a big list, which is supposed to be processed parallel afterwards. I've tried a few implementations of python's multiprocessing, but they all seem to run sequentially.

def grouper(n, iterable, padvalue=None):
  """grouper(3, 'abcdefg', 'x') -->
  ('a','b','c'), ('d','e','f'), ('g','x','x')"""
  return izip_longest(*[iter(iterable)]*n, fillvalue=padvalue)

def createRecords(givenchunk):
  for i1 in range(len(givenchunk)):
    <create somedata>
    records.append(somedata)

if __name__=='__main__':
  manager = Manager()
  parsedcdrs = manager.list([])
  records = manager.list([])

  <some general processing here which creates a shared list "parsedcdrs". Uses map to create a process "p" in some def which is terminated afterwards.>

  # Get available cpus
  cores = multiprocessing.cpu_count()

  # First implementation with map with map.
  t = multiprocessing.Pool(cores)
  print "Map processing with chunks containing 5000"
  t.map(createRecords, zip(parsedcdr), 5000)

  # Second implementation with async.
  t = multiprocessing.Pool(cores)
  for chunk in grouper(5000, parsedcdr):
    print "Async processing with chunks containing 5000"
    t.apply_async(createRecords, args=(chunk,), callback=log_result)
  t.close()
  t.join()

  # Third implementation with Process.
  jobs = []
  for chunk in grouper(5000, parsedcdr):
    t = multiprocessing.Process(target=createRecords, args=(chunk,))
    t.start()
    jobs.append(t)
  print "Process processing with chunks containing 5000"
  for j in jobs:
    j.join() 
  for j in jobs:
    j.join()

Could someone point me in te right direction?

Answer 1

Multiprocessing seems to work fine in the example above. Problem was situated in another def, resulting in slow performance.