Combine dataframes returned from python multiprocess calling function

Question

from multiprocessing import Pool

with Pool(processes=6) as p:
    p.starmap(update_tabl, zip(r))

I am using the approach outlined here: https://web.archive.org/web/20170625154652/http://nsf.github.io/2016/12/23/starmap-pattern.html to parallelize calling of a function update_table that returns a dataframe as output.

How can I concatenate all these dataframes? I could use pd.concat if I were using a for loop but not sure how to do it in parallel

Answer 1

You can do the concat on the result of the starmap:

with Pool(processes=6) as p:
    res = pd.concat(p.starmap(update_table, zip(rows)))
# do stuff with res

The concat won't happen in parallel, but once the starmap has finished.