从多个文件中读取大数据并在python中聚合数据的最快方法是什么？

Question

I have many files: 1.csv, 2.csv ... N.csv. I want to read them all and aggregate a DataFrame. But reading files sequentially in one process will definitely be slow. So how can I improve it? Besides, Jupyter notebook is used.

Also, I am a little confused about the "cost of parsing parameters or return values between python processes"

I know the question may be duplicated. But I found that most of the answers use multi-process to solve it. Multiprocess does solve the GIL problem. But in my experience(maybe it is wrong): parsing large data(like a DataFrame) as a parameter to subprocess is slower than a for loop in a single process because the procedure needs serializing and de-serializing. And I am not sure about the return of large values from the subprocess.

Is it most efficient to use a Qeueu or joblib or Ray ?

Answer 1

Reading csv is fast. I would read all csv in a list and then concat the list to one dataframe. Here is a bit of code form my use case. I find all .csv files in my path and save the csv file names in variable "results". I then loop the file names and read the csv and store it in list which I later concat to one dataframe.

data = []
for item in result:
   data.append(pd.read_csv(path))
main_df = pd.concat(data, axis = 0)

I am not saying this is the best approach, but this works great for me :)