如何将tqdm与python多处理集成

Question

I am creating a new python class where I am trying to integrate multiprocessing as well as tqdm to illustrate progress. I am going down this path because I am opening very large (>1GB) time series data files, loading into pandas, doing a groupby and then saving them in parquet format. Each datafile can take minutes to process and save. Multiprocessing speeds up the process immensely. However, I have no visibility currently on the process and I am trying to integrate tqdm.

The code below illustrates a simple example. In this code tqdm just shows how long it takes the processes to be allocated to a pool, but does not update per the actual process.

'''python

import time
import multiprocessing
from tqdm import tqdm


class test_multiprocessing(object):

    def __init__(self, *args, **kwargs):
        self.list_of_results=[]
        self.items = [0,1,2,3,4,5,6,7,8,9,10]


    def run_test(self):
        print(f'Startng test')

        for i in range(1,5,1):
            print(f'working on var1: {i}')

            p = multiprocessing.Pool()

            for j in tqdm(self.items, desc='Items', unit='items', disable=False):
                variable3=3.14159
                p.apply_async(self.worker, [i, j,variable3], callback=self.update)

            p.close()
            p.join()
            print(f'completed i = {i}')
            print(f'')

    def worker(self, var1, var2, var3):
        result=var1*var2*var3
        time.sleep(2)
        return result

    def update(self, result_to_save):
        self.list_of_results.append(result_to_save)

if __name__ == '__main__':
    test1=test_multiprocessing()
    test1.run_test()

'''

In this example the progress bar will show the work is complete almost immediately, but in reality it takes seconds

Answer 1

I found a great solution to the problem by using concurrent.futures vs multiprocessing. Dan Shiebler wrote a good blog on this and has a good example http://danshiebler.com/2016-09-14-parallel-progress-bar/

An implantation of this strategy is shown below, which solves the problem I posed earlier

import time

from tqdm import tqdm
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed

class test_multiprocessing(object):

    def __init__(self, *args, **kwargs):
        self.list_of_results=[]
        self.items = [0,1,2,3,4,5,6,7,8,9,10]


    def run_test(self):
        print(f'Startng test')

        for i in range(1,5,1):
            print(f'working on var1: {i}')

            variable_list=[]

            for j in self.items:
                variable3=3.14159
                variables = [i,j,variable3]
                variable_list.append(variables)

            with ThreadPoolExecutor(max_workers=1000) as pool:   # with ProcessPoolExecutor(max_workers=n_jobs) as pool:    
                futures = [pool.submit(self.worker, a) for a in variable_list]
                kwargs = {
                'total': len(futures),
                'unit': 'it',
                'unit_scale': True,
                'leave': True
                }

                #Print out the progress as tasks complete
                for f in tqdm(as_completed(futures), **kwargs):
                    pass

            out = []
            #Get the results from the futures. 
            for i, future in tqdm(enumerate(futures)):
                try:
                    self.update(future.result())
                except Exception as e:
                    print(f'We have an error: {e}')


    def worker(self, variables):
        result=variables[0]*variables[1]*variables[2]
        time.sleep(2)
        return result


    def update(self, result_to_save):
        self.list_of_results.append(result_to_save)

if __name__ == '__main__':
    test1=test_multiprocessing()
    test1.run_test()