How to Insert csv files to database using python_ columns

Question

This is my code

import pymysql
import csv

conn=pymysql.connect("localhost", "root","****","finance")
cursor=conn.cursor()       

print "done"
csv_data = csv.reader(file('naver_loan_output.csv'))

for row in csv_data:
    cursor.execute('INSERT INTO 'daily_new' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )' 'VALUES("%s", "%s", "%s", "%s", "%s", "%s")',row)
    conn.commit()
    cursor.close()
    print "Done"

And this is the error:

File "D:\\dropbox\\Dropbox\\lab\\7_0218\\insert_daily_new.py", line 13 cursor.execute('INSERT INTO 'daily_new' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )' 'VALUES("%s", "%s", "%s", "%s", "%s", "%s")',row) ^ SyntaxError: invalid syntax [Finished in 0.104s]

I tried a lot, but I'm not sure about the proper SQL insert query syntax. How do I get columns from csv? There are 6 columns in my csv file.

With this updated example, code works:

import pymysql
import csv

csv_data= csv.reader(file('naver_loan_output.csv'))

conn=pymysql.connect("localhost","finance")
cursor=conn.cursor()
print "Done"
for row in csv_data:
    #cursor.execute('INSERT INTO \'daily_new\' (date, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES({}, {}, {}, {}, {}, {})'.format(row[0], row[1], row[2], row[3], row[4], row[5]))
    sql="INSERT INTO  daily_n (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES('2017-01-01','2','2','2','2','2')"
    cursor.execute(sql)
    conn.commit()

cursor.close()

So, I think the for row or %s is the problem.

Answer 1

Mainly, your quotes is the issue.

1. You need to escape single quotes if larger sql string is wrapped in single quotes; or simply wrap larger string in double quotes like your updated example. And note: the SyntaxError is a Python compile error (not MySQL runtime exception).
2. For parameterized queries, do not quote the placeholder, %s .
3. MySQL (and practically for all RDMS's) do not use single quotes to enclose table and column name references as you do with 'daily_new' . Use backticks for names in MySQL. A few databases and the ANSI standard allows the double quote for object identifiers (not string literals).

Consider the following adjustment with a more efficient read process of csv file using with() as opposed to all at once as you have it with file() . And as shown with parameterization, you can prepare the sql statement once and then just bind values iteratively in loop.

strSQL = "INSERT INTO `daily_new` (`date`, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )" + \
         " VALUES(%s, %s, %s, %s, %s, %s)"

with open('naver_loan_output.csv', 'r') as f:
    csv_data = csv.reader(f)
    for row in csv_data:
        cursor.execute(strSQL, row)
        conn.commit()

cursor.close()
conn.close()

print "Done"

Answer 2

I think that you can try something like this:

query = """'INSERT INTO 'daily_new' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )' 'VALUES("""+row+""")'"""
cursor.execute(query)

Just see this code fragment from CSV File Reading and Writing doc :

>>> import csv
>>> with open('eggs.csv', 'rb') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print ', '.join(row)
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam

I hope it is useful to you or that puts you in the way.

Answer 3

There are three problems tripping you up:

1. String escape characters

In your code is that the single quote before daily_new is stopping the string and python is interpreting daily_new as a variable, which is not defined.

To solve this you should use the escape character "\\" directly before each single quote you want in the string like this:

cursor.execute('INSERT INTO \'daily_new\' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES("%s", "%s", "%s", "%s", "%s", "%s")',row)

2. Column access

The csv module returns rows as a list. To access an element in a list (or row in your case), use bracket notation. For row[0] accesses the first element (column) in a row, and row[5] accesses the 6th column.

3. String substitution

The third problem you are facing is how to pass the values into the string substitution correctly. While there are many ways to do this, an easy one is the format() method. For example: if I wanted to build a string that says "Count is 1", I could run "Count is {}".format(1) .

In your case you want to add 6 values to the string, so you add a pair of {} wherever you want a value substituted into a string and add another parameter to the format() function.

Putting it all together

So, to correct your loop code you would want something like this:

csv_data = csv.reader(file('naver_loan_output.csv'))
for row in csv_data:
    cursor.execute('INSERT INTO daily_new (date, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES ({}, {}, {}, {}, {}, {})'.format(row[0], row[1], row[2], row[3], row[4], row[5]))