I have a million records CSV file, I am using insert for single row which is taking 22 mins If i use Load data infile it takes 60 secs (But i cant use this for security purpose) The other option is multiple inserts or execute many I am confused how to use multiple inserts , execute many using for loop .I wanted a counter to check 100 rows at a time and load in database using multiple inserts and again check 100 rows and insert until End of the file Thanks in Advance
#!/usr/bin/env python
import csv
import MySQLdb
import pysftp
import smtplib
#connection to
conn = MySQLdb.connect(host="xxxxx", # The Host
user="file_auto_proc", # username
passwd="Pxx@xxx!", # password
db="xxxxx") # name of the data base
x = conn.cursor()
try:
csv_data = csv.reader(file('Disc.csv'))
for row in csv_data:
x.execute("INSERT INTO fp_feed_stg (col1,col2) VALUES(%s, %s)", row)
conn.commit()
except Exception, e:
print repr(e)
print("Not inserted")
conn.close()
I am checking the rowcount and using ciel divided by 100 and will specify the range in for loop (I did not show this condition in this code )
#!/usr/bin/env python
import csv
import MySQLdb
import pysftp
import smtplib
import itertools
#connection to
conn = MySQLdb.connect(host="xxx.xx.xxxxx.com", # The Host
user="xxxx", # username
passwd="P@xxx!", # password
db="xxxx") # name of the data base
x = conn.cursor()
csv_data = csv.reader(file('Disc1.csv'))
a=[]
i=0
k=0
for i in range(0,200):
for row in itertools.islice(csv_data, k, k+100):
a.append(row)
print row
print k
print k
sql="INSERT INTO fp_stg (col1,col2,col3,col4,col5,col6) VALUES(%s, %s, %s,%s, %s, %s)"
number_of_rows = x.executemany(sql, a)
a=[]
conn.commit()
#print i
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.