简体   繁体   中英

multiprocessing.pool code stuck and does not finish running

I am trying to use the pool class in multiprocessing module in python to do some data wrangling over a pandas data frame in parallel (code mentioned under 'Main code' heading below). The problem is my code gets stuck and does not finish running however small an input data frame (even as small as 10 rows) I provide to it. I also tried to run a simple example code (code mentioned under 'Pool example' heading below) and even that doesn't run.

Here is a detailed description of what i am trying to do in the code below: I have an indices dataframe which has 10 columns and 650K rows. The idea is to take the 10 values in each row of indices dataframe and for rows with those indexes from a target dataframe 'traindat', take a mean of a few of its columns . I have to do this for all rows of indices dataframe (650K).

Main code:

from multiprocessing import Pool
def func(x,i):
    dftmp=traindat.iloc[x,4:28].mean()
    return pd.DataFrame(dftmp).transpose()

pool = mp.Pool(processes=3)
new_rows = pool.map(func, [(row,idx) for idx,row in indices.iterrows()])
pool.close()
pool.join()
data_all_new = pd.concat(new_rows)

Since this code wouldn't run, I also tried the following simple code to see if pool runs at all for me. And it doesn't. Pool example:

import sys
sys.modules['__main__'].__file__ = 'ipython'
from multiprocessing import Pool
def f(x):
    return x*x

if __name__ == '__main__':
    p = Pool(5)
    print(p.map(f, [1, 2, 3]))

I don't get any errors in my code. It simply gets stuck and doesn't finish running. Please help me if you understand this issue.

Edit: I later realized the issue only happens in Windows. So editing the question to include that.

I realized this is a duplicate question late with the help of a colleague. Posting link to the original question and answer in case someone stumbles upon this: Basic parallel python program freezes on Windows

Seems like this is an issue related to IDE not configured properly.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM