Python 读取 mysql 到 csv

Question

I would like to read a mysql database in chunks and write its contents to a bunch of csv files.

While this can be done easily with pandas using below:

df_chunks = pd.read_sql_table(table_name, con, chunksize=CHUNK_SIZE)

for i, df in enumerate(chunks):
    df.to_csv("file_{}.csv".format(i)

Assuming I cannot use pandas, what other alternative can I use? I tried using

import sqlalchemy as sqldb
import csv

CHUNK_SIZE = 100000
table_name = "XXXXX"


host = "XXXXXX"
user = "XXXX"
password = "XXXXX"
database = "XXXXX"
port = "XXXX"

engine = sqldb.create_engine('mysql+pymysql://{}:{}@{}:{}/{}'.format(user,password,host,port,database))
con = engine.connect()
metadata = sqldb.MetaData()

table = sqldb.Table(table_name, metadata, autoload=True, autoload_with=engine)
query = table.select()
proxy = con.execution_options(stream_results=True).execute(query)

cols = [""] + [column.name for column in table.c]
file_num = 0
while True:
    batch = proxy.fetchmany(CHUNK_SIZE)

    if not batch:
        break

    csv_writer = csv.writer("file_{}.csv".format(file_num), delimiter=',')
    csv_writer.writerow(cols)
    #csv_writer.writerows(batch) # while this work, it does not have the index similar to df.to_csv()

    for i, row in enumerate(batch):
        csv_writer.writerow(i + row) # will error here

    file_num += 1
proxy.close()

While using.writerows(batch) works fine, it does not have the index like the result you get from df.to_csv() . I would like to add the row number equivalent as well, but cant seem to add to the row which is a sqlalchemy.engine.result.RowProxy . How can I do it? Or what other faster alternative can I use?

Answer 1

Look up SELECT... INTO OUTFILE...

It will do the task in 1 SQL statement; 0 lines of Python (other than invoking that SQL).