multiprocessing Pool seems to get stuck for no apparent reason

Question

I have a python script with the following, more or less, code:

def some_function():
  pass

class SomeClass:
  def __init__(self):
    self.pool = mp.Pool(10)
  def do_smth(self):
    self.pool.map(some_function, range(10))


if __name__ == '__main__':
  cls = SomeClass()
  for _ in range(1000):
    print("*")
    cls.do_smth()

the jobs are obviously much more heavy than this, however at some point it just get stuck, in a sense that no error is reported, the terminal signals that the script is still running, but no more "*" are printed, and the CPU manager of my PC reports 3% of CPU usage, so seems like that it just "crashed" without saying nothing to nobody.

For the moment, I this it might be a memory issue (however, during the time it works, RAM stays at 70%), but I have no idea... do you have any idea?

I'm working on a Macbook pro M1 max with 24 GPUs and 32GB of RAM

Answer 1

You may need to change the start_method for new processes:

mp.set_start_method('spawn')

This is the default for macOS on Python 3.8+ but not before, and it was changed due to apparent crashes using fork as reported in this issue . Apart from that if there's any thread level locking happening it could lead to deadlocks in fork mode.