如何使用 Chunk 读取大文件（大于 10 亿）

Question

I have a dataset bigger than 1 billion rows and I would like to read it by 100k rows. First I have tried to read it with nrows as below:

df = pd.read_csv("filename.csv",nrows=100000,sep='|')

But it throws an UnicodeDecodeError as below.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 3: unexpected end of data

Then I tried it with chunksize parameter as below:

a= pd.read_csv("filename.csv", chunksize=10000,sep='|')
pd.concat(a)

But it gives same error to me. I have also tried Dask library as below but it gives same error.

import dask.dataframe as dd
df = dd.read_csv('filename.csv')

Can you please help for the solution?

Thanks in advance.

Answer 1

解决方案如下：

df = pd.read_csv("filename.csv", nrows=100000, sep="|", encoding="iso8859-9")