简体   繁体   中英

Python multiprocessing with starmap and issue with __main__

I was trying to make small multiprocessing with multiple parameters

TaskType-1:

import multiprocessing  as mp
import pandas as pd
import os,sys
print("Libs Loaded..!!! ")
listoflists = [[1,2,3,4],[4,5,6,7],[7,8,9,10],[10,11,13,14]]
listoftuples = [tuple(i) for i in listoflists]
print("Length of Tuples : ",len(listoftuples))

def map_function(combo):
    a = combo[0]
    b = combo[1]
    c = combo[2]
    print((a + b + c))
    return (a + b + c)

def doit():
    try:
        print("IN main")
        p = mp.Pool(processes=2)
        # results= p.map(map_function, listoftuples)
        results = p.starmap(map_function,listoftuples)
        print(results)
        print("Done!!")
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print(exc_type, fname, exc_tb.tb_lineno)

# if '__name__' == '__main__': 
doit()     # running without entry point

Error for TaskType-1:

Libs Loaded..!!!
Length of Tuples :  4
IN main
Libs Loaded..!!!
Length of Tuples :  4
IN main
<class 'RuntimeError'> testformp.py 20
<class 'TypeError'> testformp.py 22
Libs Loaded..!!!
Length of Tuples :  4
IN main
<class 'RuntimeError'> testformp.py 20

Not sure this runtime Error and why it goes in into doit() function multiple times. Multipeprocessing is defined within this function but here it is calling the parent function again and again..Not sure what I am missing here to understand?

TaskType-2:

if '__name__' == '__main__':
    doit()  # running from entry point

output for TaskType-2:

Libs Loaded..!!!
Length of Tuples :  4
    

It shows no errors nor it performs any task inside. Why this is so?

Making the two changes I suggested I end up with this:

import multiprocessing  as mp
import os,sys
#print("Libs Loaded..!!! ")
listoflists = [[1,2,3,4],[4,5,6,7],[7,8,9,10],[10,11,13,14]]
listoftuples = [tuple(i) for i in listoflists]
#print("Length of Tuples : ",len(listoftuples))

def map_function(*combo):
    a = combo[0]
    b = combo[1]
    c = combo[2]
    #print((a + b + c))
    return (a + b + c)

def doit():
    try:
        print("IN main")
        p = mp.Pool(processes=2)
        #results= p.map(map_function, listoftuples)
        results = p.starmap(map_function,listoftuples)
        print(results)
        print("Done!!")
    except Exception as e:
        exc_type, exc_obj, exc_tb = sys.exc_info()
        fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
        print(exc_type, fname, exc_tb.tb_lineno)

if __name__ == '__main__':
    doit()     # running without entry point

And the output is this:

IN main
[6, 15, 24, 34]
Done!!

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM