尝试导入巨大的 CSV 文件时 Python 导入错误
[英]Python importing error when tried to import huge CSV file
问题描述
I am very new to python.
我对python很陌生。 I was trying to import a csv file into python using read_csv method.我试图使用 read_csv 方法将 csv 文件导入 python。
Dataset has around 500 cols and 100 K rows.数据集大约有 500 列和 10 万行。
My efforts were futile and I am getting memory error.我的努力是徒劳的，并且出现了内存错误。 I have tried to split the file into multiple blocks still I am facing the same issue.我试图将文件拆分为多个块，但仍然面临同样的问题。
Is memory issue on my system?我的系统内存有问题吗？ is python has limitation in importing such a huge number of columns? python 在导入如此大量的列方面有限制吗？ Any quick solution ?任何快速解决方案？
1 个解决方案
解决方案1
0 2016-05-04 16:31:43
http://lethain.com/handling-very-large-csv-and-xml-files-in-python/ http://lethain.com/handling-very-large-csv-and-xml-files-in-python/
"use code that treats everything as a stream" “使用将所有内容都视为流的代码”
It might help to clarify what you are doing with the data in the CSV, as that will dictate the solution to your memory issues.它可能有助于澄清您对 CSV 中的数据所做的操作，因为这将决定您的内存问题的解决方案。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.