multiprocess pool doesnt close and join terminating the script before all the process

Question

I have created a multiprocessor application that just loop some files and compare them but for some reason the pool never close and wait to join all the process responses.

from multiprocessing import Pool
def compare_from_database(row_id, connection_to_database):
    now = datetime.now()
    connection1 = sqlite3.connect(connection_to_database)
    cursor = connection1.cursor()

    grab_row_id_query = "SELECT * FROM MYTABLE WHERE rowid = {0};".format(row_id)
    grab_row_id = cursor.execute(grab_row_id_query)
    work_file_path = grab_row_id.fetchone()[1]

    all_remaining_files_query = "SELECT * FROM MYTABLE WHERE rowid > {0};".format(row_id)
    all_remaining_files = cursor.execute(all_remaining_files_query)
    for i in all_remaining_files:
        if i[1] == work_file_path:
            completed_query = "UPDATE MYTABLE SET REPEATED = 1 WHERE ROWID = {1};".format(row_id)
            work_file = cursor.execute(completed_query)
            connection1.commit()
    cursor.close()
    connection1.close()    
    return "id {0} took: {1}".format(row_id, datetime.now()-now)

I have try it with:

def apply_async(range_max, connection_to_database):
    pool = Pool()
    for i in range_of_ids:
        h = pool.apply_async(compare_from_database, args=(i, connection_to_database))
    pool.close()
    pool.join()

Also using a context and kind of force it:

from multiprocessing import Pool
with Pool() as pool:
    for i in range_of_ids:
        h = pool.apply_async(compare_from_database, args=(i, connection_to_database))
    pool.close()
    pool.join()

Even do with context shouldn't need the close/join.

The script just submit all the jobs, I can see in task manager all the python instance and are running, the print statements inside the function do print in the console fine, but once the main script finish submitting all the functions to the pools, just ends. doesn't respect the close/join

Process finished with exit code 0

if i run the function by itself runs fine returning the string.

compare_from_database(1, connection_to_database="my_path/sqlite.db")

or in a loop works fine as well

for i in range(1, 4):
    compare_from_database(i, connection_to_database="my_path/sqlite.db")

I try using python 3.7 and 3.8 and wanted to validate it with the documentation https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.join

Anyone gotten a similar issue or any ideas what might be?

Answer 1

since you want to do all the process before proceding to the next part of the script change 'async' instead async_apply that way it force to run the process and wait for the result.