Multi-threading / Parallel Processing

Question

I have few hundreds of input files that I need to pass to a function to calculate some numbers, and write them to an output file. The function does not return any value. So, each function call is independent.

Instead of calling the function serially, I tried multiprocessing, and the performance (execution time) is not much better. Any suggestions on improving the performance is valuable. Is even multiprocessing the way to go for this problem?

import multiprocessing as mp
NumProcess = 4

def Analysis(InputFile):
    #do some calcs
    #Write results to output file
    #return nothing

FileList = ['InputFile1.csv','InputFile2.csv','InputFile3.csv',....]
pool = mp.Pool(processes=NumProcess)
temp = [pool.apply_async(Analysis, args=(File) for File in FileList]
output = [p.get() for p in temp]
pool.close()

Answer 1

Does the mutiprocssing call works at all? There is an error on this line - as you don't pass the File argument as a tuple at all.

Corrected version (note the trailing comma to ensure args is a tuple):

temp = [pool.apply_async(Analysis, args=(File,)) for File in FileList]