使用多处理时有一个全局变量是否有效？

Question

Please consider this cool setup:

from multiprocessing import Pool, cpu_count
import pandas as pd
import numpy as np

def helper(master_df):
    max_index = master_df['key'].max()
    min_index = master_df['key'].min()
    #note how slave is defined before running the multiprocessing
    return slave.iloc[min_index:max_index,]

from datetime import datetime

master = pd.DataFrame({'key': [1,2,3,4,5,6,7,8,9,10]})
slave = pd.DataFrame({'key': [1,2,3,4,5,6,7,8,9,10],
                      'value' : ['a','b','c','d','e','f','g','h','i','j']})

if __name__ == '__main__':
     startTime = datetime.now()
     p = Pool(cpu_count() - 1)
     ret_list = p.map(helper, [master.iloc[1:5,], master.iloc[5:10,]])
     print datetime.now() - startTime
     print ret_list

Essentially, I have two dataframes in memory.

As you can see in the main multiprocessing code, p.map receives, as arguments, two chunks of the master dataframe.

Then, (I imagine) each process spawned by multiprocessing will access the slave dataframe and use it (without modifying). Indeed, you can see in the helper function that each process will slice the slave dataframe and do some computation with it.

My question is: is it efficient to have a dataframe defined in the global namespace that is accessed by each process? I am not sure what happens in terms of RAM utilization (is slave duplicated in memory for each process?). That would not be a good idea, because in reality both master and slave are really big.

I guess one alternative would be to send a tuple to p.map that contains the chunked master AND the corresponding sliced slave dataframe. Not sure is this is a good idea (and how to do it properly)?

Any ideas? Thanks!

Answer 1

This surprisingly depends on the operating system, as multiprocessing is implemented differently in Windows and Linux .

• In Linux, under the hood, processes are created via a fork variant where the child process shares the same address as the parent, initially, and then performs COW (copy on write). Under Linux, I've often had child processes access a read-only global DataFrame, and everything was fine (including performance).

• In Windows, under the hood, apparently, a whole process is spun up, and you might have a performance penalty of copying the DataFrame to it (unless processing done by it is large enough to render the cost negligible), but I haven't ever used Python on Windows, so have no experience with it.

Edit

An example of using joblib with DataFrames:

import joblib
import pandas as pd

df = pd.DataFrame(dict(a=[1, 3], b=[2, 3]))

def foo(i, df):
    return df + i

from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(foo)(i, df) for i in range(10))

You could also use df as a global:

def foo(i):
    return df + i

from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(foo)(i) for i in range(10))