Python - sqlite3 sqlite3.OperationalError: near “%”: syntax error?

Question

First, before everything, I realize this question has been asked beforehand. I've looked for several hours trying to solve my problem, but I haven't been able to correctly implement a solution and I'm still running into the error.

I'm trying to INSERT into a table(would like the table name to be dynamic, but I believe this is also not allowed?) using a variable string. From the studying I've done it seems that this is not allowed / good practice because it leaves the code open to SQL Injection.

I've tried to replace %s with ? but it still returns the same error with "?" instead of "%?"

Here is the code i'm using. Most of it is credited to James Mills, I'm just trying to use the statements he makes from a CSV for sqlite3 INSERT statements, if that makes sense.

"""csv2sql

Tool to convert CSV data files into SQL statements that
can be used to populate SQL tables. Each line of text in
the file is read, parsed and converted to SQL and output
to stdout (which can be piped).

A table to populate is given by the -t/--table option or
by the basename of the input file (if not standard input).

Fields are either given by the -f/--fields option (comma
separated) or determinted from the first row of data.
"""

__version__ = "0.4"
__author__ = "James Mills"
__date__ = "3rd February 2011"

import os
import csv
import sys
import optparse
import sqlite3

USAGE = "%prog [options] <file>"
VERSION = "%prog v" + __version__

def parse_options():
    parser = optparse.OptionParser(usage=USAGE, version=VERSION)

    parser.add_option("-t", "--table",
            action="store", type="string",
            default=None, dest="table",
            help="Specify table name (defaults to filename)")

    parser.add_option("-f", "--fields",
            action="store", type="string",
            default=None, dest="fields",
            help="Specify a list of fields (comma-separated)")

    parser.add_option("-s", "--skip",
            action="append", type="int",
            default=[], dest="skip",
            help="Specify records to skip (multiple allowed)")

    opts, args = parser.parse_args()

    if len(args) < 1:
        parser.print_help()
        raise SystemExit, 1

    return opts, args

def generate_rows(f):
    sniffer = csv.Sniffer()
    dialect = sniffer.sniff(f.readline())
    f.seek(0)

    reader = csv.reader(f, dialect)
    for line in reader:
        yield line

def main():
    opts, args = parse_options()

    filename = args[0]

    if filename == "-":
        if opts.table is None:
            print "ERROR: No table specified and stdin used."
            raise SystemExit, 1
        fd = sys.stdin
        table = opts.table
    else:
        fd = open(filename, "rU")
        if opts.table is None:
            table = os.path.splitext(filename)[0]
        else:
            table = opts.table

    rows = generate_rows(fd)

    if opts.fields:
        fields = ", ".join([x.strip() for x in opts.fields.split(",")])
    else:
        fields = ", ".join(rows.next())

    for i, row in enumerate(rows):
        if i in opts.skip:
            continue

        values = ", ".join(["\"%s\"" % x for x in row])
        print "INSERT INTO %s (%s) VALUES (%s);" % (table, fields, values)

        con = sqlite3.connect("school")
        cur = con.cursor()

        cur.executemany("INSERT INTO %s (%s) VALUES (%s);", (table, fields, values))
        con.commit()
        con.close()

if __name__ == "__main__":
    main()

Here is an example of the output:

> INSERT INTO data (School Name, Summer 15, Summer 16, Summer 17) VALUES ("School One", "126", "235", "453");
Traceback (most recent call last):
  File "sniffer.py", line 103, in <module>
    main()
  File "sniffer.py", line 98, in main
    cur.executemany("INSERT INTO %s (%s) VALUES (%s);", (table, fields, values))
sqlite3.OperationalError: near "%": syntax error

The sniffer stuff is the get the names of the columns and their values in which I try and put into the SQL statement.

I've tried many things, but I haven't been able to wrap my head around a solution!

Please don't bash me! I'm new to all of this and just need a little help!

Any help is appreciated!

Answer 1

Keeping in mind the possibility of SQL-Injection Attack and making sure to sanitise your inputs you can prepare your query like this:

if opts.fields:
    fields = ", ".join([x.strip() for x in opts.fields.split(",")])
else:
    fields = ", ".join(rows.next())

qry = "INSERT INTO %s (%s) VALUES (%s);" % (table,
                                            fields,
                                            ",".join("?"*len(rows)),)

Please note that for the parameter substitution in SQLite you need to use ? s.

cur.executemany(qry, (rows,))