大文件阅读问题

Question

I'm trying to read a 13GB csv file in using the following code:

chunks=pd.read_csv('filename.csv',chunksize=10000000)
df=pd.DataFrame()
%time df=pd.concat(chunks, ignore_index=True)

I have played with the values of chunksize parameter from 10 ** 3 to 10 ** 7, but everytime I receive a MemoryError . The csv file has about 3.3 Million rows and 1900 columns.

I clearly see that I have 30+GB memory available before I start reading the file, but I'm still getting the MemoryError . How do I fix this?

Answer 1

Chunking is doing nothing in the case where you want to read everything in the file. The whole purpose of chunk is to pre-process the chunk so that you then only work with the data in which you are interested (possibly writing the processed chunk to disk). In addition, it appears that your chunk size is larger than the number of rows in your data, meaning that you are reading the whole file in one go anyhow.

As suggested by @MaxU, try sparse data frames, and also use a smaller chunk size (eg 100k):

chunks = pd.read_csv('filename.csv', chunksize=100000)  # nrows=200000 to test given file size.
df = pd.concat([chunk.to_sparse(fill_value=0) for chunk in chunks])

You may also want to consider something like GraphLab Create which uses SFrames (not limited by RAM).