[英]How can I write a large csv file using Python?
I need to extract a big amount of data(>1GB) from a database to a csv file. 我需要从数据库中提取大量数据(> 1GB)到csv文件。 I'm using this script:
我正在使用此脚本:
rs_cursor = rs_db.cursor()
rs_cursor.execute("""SELECT %(sql_fields)s
FROM table1""" % {"sql_fields": sql_fields})
sqlData = rs_cursor.fetchall()
rs_cursor.close()
c = csv.writer(open(filename, "wb"))
c.writerow(headers)
for row in sqlData:
c.writerow(row)
The problem comes when is writing the file the system runs out of memory. 问题是在写入文件时系统内存不足。 In this case, is there any other and more efficient way to create a large csv file?
在这种情况下,还有其他更有效的方法来创建大型csv文件吗?
psycopg2
(which OP uses) has a fetchmany
method which accepts a size
argument. psycopg2
(OP使用)具有fetchmany
方法,该方法接受size
参数。 Use it to read a certain number of lines from the database. 使用它从数据库中读取一定数量的行。 You can expirement with the value of
n
to balance between run-time and memory usage. 您可以使用
n
值终止,以在运行时和内存使用之间取得平衡。
fetchmany
docs: http://initd.org/psycopg/docs/cursor.html#cursor.fetchmany fetchmany
文档: http : fetchmany
rs_cursor = rs_db.cursor()
rs_cursor.execute("""SELECT %(sql_fields)s
FROM table1""" % {"sql_fields": sql_fields})
c = csv.writer(open(filename, "wb"))
c.writerow(headers)
n = 100
sqlData = rs_cursor.fetchmany(n)
while sqlData:
for row in sqlData:
c.writerow(row)
sqlData = rs_cursor.fetchmany(n)
rs_cursor.close()
You can also wrap this with a generator to simplify the code a little bit: 您还可以使用生成器包装它,以简化代码:
def get_n_rows_from_table(n):
rs_cursor = rs_db.cursor()
rs_cursor.execute("""SELECT %(sql_fields)s
FROM table1""" % {"sql_fields": sql_fields})
sqlData = rs_cursor.fetchmany(n)
while sqlData:
yield sqlData
sqlData = rs_cursor.fetchmany(n)
rs_cursor.close()
c = csv.writer(open(filename, "wb"))
c.writerow(headers)
for row in get_n_rows_from_table(100):
c.writerow(row)
Have you tried fetchone()? 您是否尝试过fetchone()?
rs_cursor = rs_db.cursor()
rs_cursor.execute("""SELECT %(sql_fields)s
FROM table1""" % {"sql_fields": sql_fields})
c = csv.writer(open(filename, "wb"))
c.writerow(headers)
row = rs_cursor.fetchone()
while row:
c.writerow(row)
row = rs_cursor.fetchone()
rs_cursor.close()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.