简体   繁体   中英

insert csv file with multiple insert in mysql python

I have a million records CSV file, I am using insert for single row which is taking 22 mins If i use Load data infile it takes 60 secs (But i cant use this for security purpose) The other option is multiple inserts or execute many I am confused how to use multiple inserts , execute many using for loop .I wanted a counter to check 100 rows at a time and load in database using multiple inserts and again check 100 rows and insert until End of the file Thanks in Advance

#!/usr/bin/env python  
import csv
import MySQLdb
import pysftp
import smtplib

#connection to 
conn =  MySQLdb.connect(host="xxxxx", # The Host
                      user="file_auto_proc", # username
                      passwd="Pxx@xxx!", # password
                      db="xxxxx") # name of the data base

x = conn.cursor()
try:
    csv_data = csv.reader(file('Disc.csv'))
    for row in csv_data:

        x.execute("INSERT INTO fp_feed_stg (col1,col2) VALUES(%s, %s)", row)
    conn.commit()


except Exception, e: 
    print repr(e)
    print("Not inserted")



conn.close()

I am checking the rowcount and using ciel divided by 100 and will specify the range in for loop (I did not show this condition in this code )

#!/usr/bin/env python  
import csv
import MySQLdb
import pysftp
import smtplib
import itertools

#connection to 
conn =  MySQLdb.connect(host="xxx.xx.xxxxx.com", # The Host
                      user="xxxx", # username
                      passwd="P@xxx!", # password
                      db="xxxx") # name of the data base

x = conn.cursor()


csv_data = csv.reader(file('Disc1.csv'))    
a=[]    
i=0
k=0

for i in range(0,200):      
    for row in itertools.islice(csv_data, k, k+100):
        a.append(row)
        print row
        print k
    print k

        sql="INSERT INTO fp_stg (col1,col2,col3,col4,col5,col6) VALUES(%s, %s, %s,%s, %s, %s)"
    number_of_rows = x.executemany(sql, a)
    a=[]
    conn.commit()

#print i

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM