Python importing error when tried to import huge CSV file

Question

I am very new to python. I was trying to import a csv file into python using read_csv method.

Dataset has around 500 cols and 100 K rows.

My efforts were futile and I am getting memory error. I have tried to split the file into multiple blocks still I am facing the same issue.

Is memory issue on my system? is python has limitation in importing such a huge number of columns? Any quick solution ?

Answer 1

http://lethain.com/handling-very-large-csv-and-xml-files-in-python/

"use code that treats everything as a stream"

It might help to clarify what you are doing with the data in the CSV, as that will dictate the solution to your memory issues.