当我通过 NFS 处理 GB 大小的文件时，如何在 Python 中优化文件 I/O？

Question

I'm manipulating several files via nfs, due to security concerns. The situation is very painful to process something due to slow file I/O. Followings are descriptions of the issue.

• I use pandas in Python to do simple processing on data. So I use read_csv() and to_csv() frequently.
• Currently, writing of a 10GB csv file requires nearly 30 mins whereas reading consumes 2 mins.
• I have enough CPU cores (> 20 cores) and memory (50G~100G).
• It is hard to ask more bandwidth.
• I need to access data in column-oriented manner, frequently. For example, there would be 100M records with 20 columns (most of them are numeric data). For the data, I frequently read all of 100M records only for 3~4 columns' value.
• I've tried with HDF5, but it constructs a larger file and consumes similar time to write. And it does not provide column-oriented I/O. So I've discarded this option.
• I cannot store them locally. It would violate many security criteria. Actually I'm working on virtual machine and file system is mounted via nfs.
• I repeatedly read several columns. For several columns, no. The task is something like data analysis.

Which approaches can I consider? In several cases, I use sqlite3 to manipulate data in simple way and exports results into csv files. Can I accelerate I/O tasks by using sqlite3 in Python? If it provide column-wise operation, it would be a good solution, I reckon.

Answer 1

two options: pandas hdf5 or dask.

1. you can review hdf5 format with format='table'.

HDFStore supports another PyTables format on disk, the table format. Conceptually a table is shaped very much like a DataFrame, with rows and columns. A table may be appended to in the same or other sessions. In addition, delete and query type operations are supported. This format is specified by format='table' or format='t' to append or put or to_hdf.

2. you can use dask read_csv. it read data only when execute()

For purely improve IO performance, i think hdf with compress format is best.