Multiprocessing Pool getting stuck

Question

I don`t have much experience with multiprocessing but thus far it seems to hate me. I am trying to run the most simplest of examples, and yet it gets stuck and I have to restart my kernel.

My example:

import multiprocessing as mp

aa = [x for x in range(3,16)]

def f(x):
    return x**2

if __name__ == '__main__':
    with mp.Pool(processes = 4) as p:
        res = p.map(f, aa)

print(res)

It worked just fine up until I experimented with using class methods and atributes inside of the pool.map() . I think that it broke after I tried to execute this:

class clasy():
    def __init__(self, a, b):
        self.a = a
        self.b = b

    def gen(self):
        for i in range(self.a, self.b):
            yield i

    def dodo(self):
        gg = gen()
        with mp.Pool() as pool:
            self.res = pool.map(f, gg)
        return self.res

paralel = clasy(3,16)
print(paralel.dodo())

What went wrong and can it be fixed?

And further more, is it possible to use class method for generator inside of the pool.map(), can it be used inside a class. Or should the pool, function and iterator be top level objects?

Answer 1

You need to either pass the function f to the class clasy or make it as an instance of the class. For example, the following should work:

class clasy():
    def __init__(self, a, b):
        self.a = a
        self.b = b

    def gen(self):
        for i in range(self.a, self.b):
            yield i

    def f(self, x):
        return x**2

    def dodo(self):
        gg = self.gen()
        with mp.Pool(processes = 4) as pool:
            self.res = pool.map(self.f, gg)
        return self.res

paralel = clasy(3,16)
print(paralel.dodo())

Out[1]:
[9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225]