Multiprocessing in Python Issue

Question

I'm trying multiprocessing in python but can't seem to get it to work.

The input file is as follows:

And the code is as follows:

import pandas as pd
import multiprocessing
import time
import datetime


start_time = datetime.datetime.now()
df_main = []
df_main = pd.read_csv("data.csv")
df_file = []

def growth_calculator(Type):
    values = [Type]
    global df_temp, df_file
    df_temp = df_main[df_main.Type.isin(values)]
    df_temp = df_temp[['Company', 'Type']]
    print(df_temp)
    time.sleep(10)

if __name__ == '__main__':
    multiprocessing.Process(target=growth_calculator('Quarterly'))
    multiprocessing.Process(target=growth_calculator('Annual'))
    multiprocessing.Process(target=growth_calculator('Monthly'))
    end_time = datetime.datetime.now()

print("Time Taken -", end_time-start_time)

The output should take around 10-11 seconds, but it's taking 30 seconds. So, clearly, multiprocessing isn't working.

Could you please point me to the right direction?

Thanks in advance!

Answer 1

you need to pass target arguments as args= keyword for the Process init (see https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process ). Otherwise your function is evaluated before instantiating process, which leads to single-process performance.

Something like this:

import pandas as pd
import multiprocessing
import time
import datetime


start_time = datetime.datetime.now()


def growth_calculator(Type):

    print(Type)
    time.sleep(10)

if __name__ == '__main__':
    p1 = multiprocessing.Process(target=growth_calculator,args=('Quarterly',))
    p2 = multiprocessing.Process(target=growth_calculator,args=('Annual',))
    p3 = multiprocessing.Process(target=growth_calculator,args=('Monthly',))
   
    p1.start()
    p2.start()
    p3.start()
    print('started')
    p1.join()
    p2.join()
    p3.join()
    end_time = datetime.datetime.now()

    print("Time Taken -", end_time-start_time)