Python：在多处理进程之间共享大型数据帧

Question

I am new to Python and multiprocessing. I have to parse 2 large xml (around 6GBs) files into 2 dataframes. both files can be processed independently.

As far as i learnt in Python, i can do that with multiprocessing. So Process 1 for parsing xml1 file and load it into dataframe Process 2 for parsing xml2 file and load it into dataframe

Now i want to use dataframe generated by process 1 into process 2 Can anyone tell me which is the best way to implement this ? My main concern is sharing dataframe between processes.

Regards Vipul

Answer 1

You are using multiple processes mainly because you want to read the data in parallel. Once it has been read in there doesn't seem to be much reason why you should continue with two processes. Ie your processes should read the data, then terminate and let the main process continue.

I would however recommend using multi-threading instead of multi-processing . The differences are not always noticeable, but multi-threading will make it simpler to share global variables between your main thread and the subthreads (I'll explain this below). Another advantage of multi-threading is that it doesn't cause the whole application to crash if a single thread fails. That is not the case with multi-processing. See more here: http://net-informations.com/python/iq/multi.htm

It takes some time to get used to how parallelism works in Python, and one of the main considerations you have to pay attention to is how to ensure what you are doing is thread-safe. Usually the mechanism you use for passing data to and from a thread is a queue. This ensures that only one thread is accessing the same object at any given time.

That being said, in your simple example you could simply define two global variables and start two threads that each read data into one of these global variables (ie no sharing of variables across threads). You also have to tell your main thread to wait until both your threads have completed before continuing as otherwise the main thread could try to access the data while the subthreads are still working on them. (Again, normally you would adopt a queue-based strategy to avoid this problem, but you don't necessarily need that here).

Here is some example code:

import threading
import pandas as pd
import time

def get_df_1():
    #set the scope of the variable to "global", meaning editing it here, it is edited globally
    global df_1 

    # read in your xml file here (for the example I simply create some dummy data)
    data = [['tom', 10], ['nick', 15], ['juli', 14]]
    df_1 = pd.DataFrame(data, columns=['Name', 'Age'])

    # wait five seconds (for illustration purposes to simulate  working time)
    time.sleep(5)
    print("df_1 fetched")

def get_df_2():
    global df_2
    data = [['tom', 176], ['nick', 182], ['juli', 167]]
    df_2 = pd.DataFrame(data, columns=['Name', 'Height'])
    time.sleep(5)
    print("df_2 fetched")

df_1 = None
df_2 = None

#define threads
t1 = threading.Thread(target=get_df_1)
t2 = threading.Thread(target=get_df_2)

# start threads
t1.start()
t2.start()

#this will print immediately
print("Threads have been started")

# wait until threads finish
t1.join()
t2.join()

#this will only print after the threads are done
print("Threads have finished")

print(df_1)
print(df_2)