Graceful cleanup for multiprocess pool python

Question

I am running a multiprocessing pool, mapped over a number of inputs. My worker processes have an initialization step that spins up a connection to selenium and a database. When the pool finishes its job, what is the graceful way to close these connections rather than just relying on python's memory management and del definitions?

EDIT:

class WebDriver():
  def close():
    //close logic

  def __del__():
    self.driver.close()

def init():
  global DRIVER
  DRIVER=WebDriver()

def shutdown():
  DRIVER.close()

if __name__=='__main__':
  with multiprocessing.Pool(initializer=init) as pool:
    pool.map(some_function, some_args)

Because some_args is large, I only want to call shutdown when the worker processes have no other jobs to do. I don't want to close / reopen connections to my database until everything is done.

As of right now, I would expect the memory manager to call __del__ if the worker process shutsdown, but I don't know if it does occur. I've gotten strange scenarios where it hasn't been called. I'm hoping to better understand how to manage shutdown.

Answer 1

I think you have a good chance of closing your drivers if you first wait for your pool processes to terminate and then force a garbage collection:

if __name__=='__main__':
    with multiprocessing.Pool(initializer=init) as pool:
        pool.map(some_function, some_args)
        # Wait for all tasks to complete and all processes to terminate:
        pool.close()
        pool.join()
        # Processes should be done now:
        import gc
        gc.collect() # ensure garbage collection