并行化此Python结构的正确方法

Question

I'm writing some code in Python and hope that I can speed it up by sharing it between different processes. I have the following program substructure in Python 3.5 on Linux:

class Foo(fromMother):

   def __init__(self, size):
      fromMother.__init__()
      self.data = self.__createData(size)
      self.properGuess = 0

   def __createData(self, size):
      """Returns some dict of n x m numpy arrays."""

   def doStuff(self, level):
      """Calculates CPU-heavy, fully serial operation.
        The proper guess is some sort of start value."""
      a, b = foo(level, self.properGuess)
      self.properGuess = a
      return b



if __name__ == "__main__":
   foo = Foo(someSize)
   array = []
   for i in range(manyValues):
       array.append(foo.doStuff(i))

This is very roughly what my code does and I would like to find a nice way to introduce some parallel structure to use multiple cores. doStuff() can not be parallelized and is some kind of optimization problem, hence the starting value. It is not essential that this is the absolutely latest value, the function still gets executed fairly well at step i with a starting point calculated at i - 10, so it's completely okay to update this value 'every now and then'. But it can't be left out, the value of step i+1 is close to i and I know that the problem is not globally but only locally convex. Staying that close has given me quite stable results.

Unfortunately, breaking up the inheritance structure would be quite annoying - it's possible but should be the last resort. I wanted to use Python's own multiprocessing library but I'm a little confused with the shared memory etc, since this is not simply sharing some sort of array but having it in an object and using methods on it. I have only done parallel programming on MATLAB and this just on a very basic level.

Could someone point me a way to this. I'm thinking about this for two days already and I'm not sure how to achieve this nicely without breaking and "uglyfying" my entire structure.

Answer 1

You can use a process pool

from multiprocessing import Pool

if __name__ == "__main__":
   foo = Foo(someSize)
   p = Pool(5)
   results = p.map(foo.doStuff, range(manyValues))

But, you need to think about some things first, is your class sharable?, will it mutate?, etc...