python sqlalchemy可以使用变量构造插入语句吗（对于mysql数据库）

Question

Thanks in advance for advice on this problem...

I am trying to create a python script to import a set of CSVs into a mysql database.

Each CSV filename matches the destination table. The first row of each CSV matches the fields of the table. Each CSV / table has a different number of fields, field names, etc.

The problem I am having is with this line (full code below)

ins = table_name.insert().values(temp_variable_name)

where I want to dynamically update the destination table (table_name) and the insert command (temp_variable_name).

So when reading the labels.csv file, this should produce

ins = labels.insert().values(id_label=d[0], label_name=d[1])

and when reading the company.csv file, this should produce

ins = company.insert().values(id_company=d[0], company_name=d[1], ticker=d[2])

The problem is if I generate a string,

temp_variable_name = 'id_company=d[0], company_name=d[1], ticker=d[2]'

I end up getting a 'str' object has no attribute 'items' error.

Is there any way to dynamically generate an insert command for an SQL statement?

Portion of the script below:

# files list contains a list of all of the files in the directory
# we read in CSVs, isolate the first row to determine table field names
# the rest of the data should then be imported into the table with the corresponding name as the CSV

for f in files:
    if '.csv' in f :

        # read in each CSV file

        # these are a Class / Function I've set up to read files
        x = Read_Files()  
        data = x.read_file_lines_strip(path, f)

        temp = data[0].replace('"','') # get rid of quotation marks from the data

        table_header_list = temp.split('|') # get the first row, which is the table field names

        variable_name ='' # this is used to construct the insert into table string

        for x in xrange (0, len(table_header_list)) :
            if x == 0 :
                variable_name = variable_name + table_header_list[0] + '=d[0]'
            elif x == len(table_header_list) :
                variable_name = variable_name + table_header_list[x] + '=d[' + str(x) + ']'
            else :
                variable_name = variable_name + ', ' + table_header_list[x] + '=d[' + str(x) + ']'

        table_name = f.replace('.csv','') # remove the .csv from filename to isolate the file name, which is the same as table name

        # data from file

        for data_line in data[1:] :
            data_line = data_line.replace('"', '') # remove quotation marks
            d = data_line.split('|') # split the line which is delimited by a |

            # used to construct the final insert string
            for x in xrange(0, len(table_header_list)) :
                if x == 0 :
                    temp_variable_name = variable_name.replace('d[0]', d[0])
                else :
                    temp_variable_name = temp_variable_name.replace('d[' + str(x) + ']', d[x])


            try:
                # table name is the table to insert into, via the CSV filename
                # temp_variable_name is the insert string, such as 'id_company=d[0], company_name=d[1], ticker=d[2]'
                ins = table_name.insert().values(temp_variable_name)
                result = conn.execute(ins)
            except Exception, e :
                print 'error : ' + str(e)

Answer 1

You can do this with Insert objects and the csv module which makes it easy with theDictReader class. Here's an example for the company table:

import csv
from sqlalchemy import create_engine
from sqlalchemy.sql import table, column

NULL_FIELD_VALUE = r'\N'
DB_CONNECT = 'sqlite:///company.db'
engine = create_engine(DB_CONNECT, echo=True)
conn = engine.connect()

with open('company.csv') as csvfile:
    reader = csv.DictReader(csvfile, delimiter='|')
    insert_table = table('company',
                         *[column(field) for field in reader.fieldnames])
    insert_dict = [{k: None if v == NULL_FIELD_VALUE else v
                        for k,v in row.items()}
                            for row in reader]
    conn.execute(insert_table.insert(), insert_dict)