流式处理大型csv文件的最快方法是什么？

Question

I've compared the built-in csv reader with Pandas's read_csv . The former is significantly slower. However, I have a need to stream csv files due to memory limitation. What streaming csv reader that is as fast or almost as fast as Pandas?

Answer 1

在pandas.read_csv中，您可以使用“ chunksize”选项，如果这样做，pandas返回的对象将是一个迭代器（TextFileReader类型），当迭代结束时，它将返回一个读取number_of_rows <= chunksize的DataFrame（我没有意识到该选项存在，直到我阅读源代码...）。