Python multiprocessing finish the work correctly, but the processes still alive (Linux)

Question

I use python multiprocessing to compute some sort of scores on DNA sequences from a large file. For that I write and use the script below. I use a Linux machine with 48 cpu in python 3.8 environment.

Th code work fine, and terminate the work correctly and print the processing time at the end.

Problem : when I use the htop command, I find that all 48 processes are still alive. I don't know why, and I don't know what to add to my script to avoid this.

import csv
import sys
import concurrent.futures
from itertools import combinations
import psutil
import time


nb_cpu = psutil.cpu_count(logical=False)


def fun_job(seq_1, seq_2): # seq_i : (id, string)
    start = time.time()
    score_dist = compute_score_dist(seq_1[1], seq_2[1])
    end = time.time()

    return seq_1[0], seq_2[0], score_dist, end - start # id seq1, id seq2, score, time


def help_fun_job(nested_pair):
    return fun_job(nested_pair[0], nested_pair[1])


def compute_using_multi_processing(list_comb_ids, dict_ids_seqs):
    start = time.perf_counter()

    with concurrent.futures.ProcessPoolExecutor(max_workers=nb_cpu) as executor:
        results = executor.map(help_fun_job,
                               [((pair_ids[0], dict_ids_seqs[pair_ids[0]]), (pair_ids[1], dict_ids_seqs[pair_ids[1]]))
                                for pair_ids in list_comb_ids])

    save_results_to_csv(results)

    finish = time.perf_counter()

    proccessing_time = str(datetime.timedelta(seconds=round(finish - start, 2)))
    print(f' Processing time Finished in {proccessing_time} hh:mm:ss')

def main():
    print("nb_cpu in this machine : ", nb_cpu)

    file_path = sys.argv[1]

    dict_ids_seqs = get_dict_ids_seqs(file_path)

    list_ids = list(dict_ids_seqs)  # This will convert the dict_keys to a list
    list_combined_ids = list(combinations(list_ids, 2))
    
    compute_using_multi_processing(list_combined_ids, dict_ids_seqs)


if __name__ == '__main__':
    main()

Thank you for your help.

Edit : add the complete code for fun_job (after @Booboo answer)

from Bio import Align

def fun_job(seq_1, seq_2): # seq_i : (id, string)
    start = time.time()

    aligner = Align.PairwiseAligner()
    aligner.mode = 'global'
    score_dist = aligner.score(seq_1[1],seq_2[1])    

    end = time.time()

    return seq_1[0], seq_2[0], score_dist, end - start # id seq1, id seq2, score, time

Answer 1

When the with ... as executor: block exits, there is an implicit call to executor.shutdown(wait=True) . This will wait for all pending futures to to be done executing " and the resources associated with the executor have been freed ", which presumably includes terminating the processes in the pool (if possible?). Why your program terminates (or does it?) or at least you say all the futures have completed executing, while the processes have not terminated is a bit of a mystery. But you haven't provided the code for fun_job , so who can say why this is so?

One thing you might try is to switch to using the multiprocessing.pool.Pool class from the multiprocessing module. It supports a terminate method, which is implicitly called when its context manager with block exits, that explicitly attempts to terminate all processes in the pool:

#import concurrent.futures
import multiprocessing
... # etc.

def compute_using_multi_processing(list_comb_ids, dict_ids_seqs):
    start = time.perf_counter()

    with multiprocessing.Pool(processes=nb_cpu) as executor:
        results = executor.map(help_fun_job,
                               [((pair_ids[0], dict_ids_seqs[pair_ids[0]]), (pair_ids[1], dict_ids_seqs[pair_ids[1]]))
                                for pair_ids in list_comb_ids])

    save_results_to_csv(results)

    finish = time.perf_counter()

    proccessing_time = str(datetime.timedelta(seconds=round(finish - start, 2)))
    print(f' Processing time Finished in {proccessing_time} hh:mm:ss')