Python multiprocessing program not running to the end

Question

I am new to multiprocessing in python.

Basically my problem scenario is that I want to run my python script parallely on set of tables say 2 tables.

Here my python script reads the data from each of the tables parallely and then write the data from each of these tables into another table.

I have written the following code snippet to create a multiprocess python script. However, when I run the script it does not complete and neither does it throw any error message.

count = multiprocessing.cpu_count()
pool = multiprocessing.Pool(processes=count)
args = [ ('yelp','localhost:9160','cassa1','flight88'), ('yelp','localhost:9160','cassa1','flight96') ]
for a in args:
    print a
    pool.apply_async(user_input,a)

Appreciate help on this as I am confused as well as stuck here.

Answer 1

Your script exits before child processes finish their tasks. Add at the end:

pool.close() # no more tasks
pool.join()  # wait for the remaining tasks to complete

Also, you could use pool.imap*() methods instead:

from multiprocessing import Pool

def safe_user_input(args):
    try:
         return user_input(*args), None
    except Exception as e:
         return None, str(e)

if __name__=="__main__":       
   tables = [
        ('yelp','localhost:9160','cassa1','flight88'),
        ('yelp','localhost:9160','cassa1','flight96')
   ]

   pool = Pool() # use all available CPUs
   for result, error in pool.imap_unordered(safe_user_input, tables):
       if error is None: # no error
          print(result)