将 kwargs 传递给带有用于 Pandas Dataframe 的 multiprocessing.pool.map 的函数

Question

I have written a function that takes in a dataframe with keywords arguments as shown below:

df1 = df1.apply(add_data_ip2, axis=1, result_type="expand")

The process takes 20 minutes. the function add_data_ip2 takes in a dataframe and reads the stock symbol under the "Ticker" column and does an api call to retrieve financial info and manipulates the data through math to calculate a score. The score is under saved in the "Score" column of the same df. The function returns the same dataframe.

The df contains approximately 1500 ticker symbols and I am trying to run the following parallel processing code to reduce the waiting time, but with no luck. The function keeps running with no indication of any output. Can anyone advise what is the problem? Is there anything wrong with the way I am passing in the kwargs into the function. Have tried searching stackoverflow for answers with no luck. Appreciate the help.

from functools import partial
mapfunc = partial(add_data_ip2, axis=1, result_type="expand")

p = mp.Pool(mp.cpu_count())
df1= p.map(mapfunc, df1)
df1

Another alternative block also doesn't give any output either.

from multiprocessing import  Pool
    
def parallelize_dataframe(df, func, n_cores=4):
        df_split = np.array_split(df, n_cores)
        pool = Pool(n_cores)
        df = pd.concat(pool.map(func, df_split))
        pool.close()
        pool.join()
        return df

train = parallelize_dataframe(df1, mapfunc)'

Answer 1

As for me your partial is wrong.

You can't send df to partial partial(add_data_ip2,...) and run as df.apply(add_data_ip2, ...) because partial will try to run it as add_data_ip2(..., df)

Other problem: axis=1, result_type="expand" are parameters for df.apply() but partial will run it as add_data_ip2(..., axis=1, result_type="expand")

As for me you should define normal function

def mapfunc(dfx):
    return dfx.apply(add_data_ip2, axis=1, result_type="expand")

or lambda

mapfunc = lambda dfx: dfx.apply(add_data_ip2, axis=1, result_type="expand")

But as I know Pool can't work with lambda because it has to save function and data in pickle and later process has to read it - but pickle can't save lambda

---

Code which I used for tests

import pandas as pd
from functools import partial
from multiprocessing import Pool
import numpy as np

data = {
    'A': [1,2,3], 
    'B': [4,5,6], 
    'C': [7,8,9]
}

df = pd.DataFrame(data)
print('--- original ---')
print(df)

def add_data_ip2(row):
    row['A'] += 10
    row['B'] += 100
    row['C'] += 1000
    return row

# --- test 1 ---
#new_df = df.apply(add_data_ip2, axis=1, result_type="expand")
#print(new_df)  # OK

# --- test 2 ---
#mapfunc = partial(add_data_ip2, axis=1, result_type="expand")
#new_df = mapfunc(df)  # ERROR
#print(new_df)
    
# --- test 3 ---
#mapfunc = lambda df: df.apply(add_data_ip2, axis=1, result_type="expand")
#new_df = mapfunc(df)  # OK
#print(new_df)

# --- test 4 ---
def mapfunc(df):
    return df.apply(add_data_ip2, axis=1, result_type="expand")

new_df = mapfunc(df)  # OK
print('--- mapfunc ---')
print(new_df)

# --- test Pool ---

p = Pool()

parts = np.array_split(df, 4)

results = p.map(mapfunc, parts)

print('--- Poll results ---')
for item in results:
    print(item)
    print('---')
    
print('--- concat new df ---')    
new_df = pd.concat(results)
print(new_df)