Python 多线程按顺序执行线程 Pandas Dataframe

Question

Is there a way to execute threads in a specific order ? I am familiar with the commands ".wait()" and ".notifyAll()", however it does not seem to work when all threads are targeting to a single function. The code below should write the csv file in this order: df1, df2, df3, df4.

---

import threading
import pandas as pd


df1 = pd.DataFrame(columns=["col1","col2","col3"])
df2 = pd.DataFrame(columns=["col1","col2","col3"])
df3 = pd.DataFrame(columns=["col1","col2","col3"])
df4 = pd.DataFrame(columns=["col1","col2","col3"])


def function(df):
    ###webscraping, compile web data to dataframe
    df.to_csv('output.csv', mode='a')


if __name__ == '__main__':
    t1 = threading.Thread(target=function, args=(df1,))
    t2 = threading.Thread(target=function, args=(df2,))
    t3 = threading.Thread(target=function, args=(df3,))
    t4 = threading.Thread(target=function, args=(df4,))
    t1.start()
    t2.start()
    t3.start()
    t4.start()

---

I want all dataframes to wait inside "function()" until they can execute in order . With multithreading, threads like to "race each other" and can fall out of order executed . Although multithreading is a good performance enhancing tool, its' downfall comes into play when order matters.

Example of Simplicity: If thread 4 finishes compiling its' dataframe, it needs to wait for the first 3 threads to compile its' corresponding dataframe and upload to the csv file until thread 4 can upload.

As always, thanks in advance!!

Answer 1

To solve your problem in a clean way, you probably want to be using concurrent.futures instead of threading , hopefully you're on python3.2+.

To do so, you want to create a list of your arguments to the function in the order you need them to write arglist = [df1, df2, ...] , and then do something like

from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=len(arglist)) as ex:
    results = ex.map(function, arglist)
for res in results:
    res.to_csv(..., mode='a')

To be honest, you should really try to use concurrent.futures for everything related to threading or multiprocessing.

It appears I read the question wrong the first time. I'll leave my previous answer here for people who google

You can use a lock (see https://docs.python.org/3/library/threading.html#lock-objects ), and then call lock.acquire() before writing to csv and then lock.release() afterwards. This will do exactly what you want.

Although in my opinion this is not ideal, instead I would suggest returning the dataframes from each thread and just writing them all at the end.

Your code would simply look like ``` lock = threading.Lock()

def function(args): # web stuff with lock: df.to_csv(...) ```