推荐使用python读取100GB .csv.gz文件

Question

I have a ".csv.gz" file that is 100GB large in a remote linux. I definitely do not want to unzip it because the size would reach to 1T.

I am looking online for reading files. I saw on suggestion here

python: read lines from compressed text files

gzip? pandas? iterator?

My mentor suggested to pip the data after unzip it.

I would also need to consider the memory. So readlines() is definitely not my consideration.

I wonder if anyone has an optimal solution for this because the file is really large and it would take me a lot of time to just do anything.

Answer 1

您可以将文件大块for line in sys.stdin: ...到python中，并像for line in sys.stdin: ...的一行一样for line in sys.stdin: ...处理它for line in sys.stdin: ... ：

zcat 100GB.csv.gz | python <my-app>

Answer 2

read the lines one by one by doing:

import sys

for line in sys.stdin:
    do_sth_with_the_line(line)

You call this python script with:

zcat | python_script.py