使用 Python 读取位于 S3 (AWS) 上的大型 CSV 文件（10 M+ 条记录）的最有效方法是什么？

Question

I've been trying to find the fastest way to read a large csv file ( 10+ million records) from S3 and do a couple of simple operations with one of the columns ( total number of rows and mean). I have ran a couple of tests, and the fastest so far was creating a dask dataframe, but I am wondering if there is any other alternative out there that may make things even faster.

Any suggestions? Thanks!

Test 1. Pandas read csv: 92.36531567573547 seconds

start_time = time.time()
s3 = boto3.client('s3')
path =my_csvS3
use_column=['tip_amount']
df= pd.read_csv(path,usecols=use_column)
print(df.count)
print (df["tip_amount"].mean())
print("%s seconds" % ((time.time())-(start_time)))

Test 2 Pandas read csv in chunks: 78.15214204788208 seconds

import time
start_time = time.time()
tp = pd.read_csv(path, usecols=use_column, iterator=True, chunksize=5000000)  # gives TextFileReader
df = pd.concat(tp, ignore_index=True)
print(df.count)
print (df["tip_amount"].mean())
print("%s seconds" % ((time.time())-(start_time)))

Test 3 dask dataframe: 54.183971881866455 seconds

import dask.dataframe as dd
import time
start_time = time.time()
s3 = boto3.client('s3')
df = dd.read_csv(path)
df = df['tip_amount'] 
cols=['tip_amount']
dfp = df.compute()
print(len(dfp))
print (dfp.mean())
print("%s seconds" % ((time.time())-(start_time)))

Answer 1

This line

dfp = df.compute()

is an antipattern for dask. You split up the load, btu then you form a single large dataframe in memory by concatenation. You would do better to compute what you want on the original chunks (note that len is special in python, so this is less tidy than it would be

dask.compute(df.shape[0], df.mean())

Also, you might find better performance with the distributed scheduler, even on a single machine, for some workloads. In this case, I believe most of the work is GIL-free, which is the critical consideration. Still, worth measuring the difference.

Additional: if you really only want one column, specify this in your read_csv - this goes for all backends!