Dynamically import columns from csv into database table in sqlite3 and python

Question

How do I take an arbitrary csv file and create a new table with the specified columns in it using python? Let's say I have many different csv files with different column names and I want to create a table for each individual file using python.

Answer 1

In the following example I presume that the files contain a first line that defines column names. Otherwise, you can just use a fixed list of names. The solution I outline can be refined to test the actual data values to make inferences about their type, but I assume that initially you will be happy with a solution that just defines a varchar field for each column in the CSV file (sqlite stores everything as character data anyway).

In [54]: f = open("/Users/sholden/test.csv", 'rU')

In [55]: reader = csv.reader(f)

In [56]: names = reader.next()

In [57]: names
Out[57]: ['First', 'Second', 'Third']

In [65]: import sqlite3 as db

In [66]: conn = db.connect(":memory:")

In [67]: curs = conn.cursor()

In [68]: sql = """create table x (\n""" +\
      ",\n".join([("%s varchar" % name) for name in names])\
      + ")"

In [69]: sql
Out[69]: 'create table x (\nFirst varchar,\nSecond varchar,\nThird varchar)'

In [70]: curs.execute(sql)
Out[70]: <sqlite3.Cursor at 0x101f2eea0>

In [71]: for line in reader:
    curs.execute("""INSERT INTO x (First, Second, Third)
                    VALUES (?, ?, ?)""", tuple(line))
   ....:     

In [72]: curs.execute("SELECT * FROM x")
Out[72]: <sqlite3.Cursor at 0x101f2eea0>

In [73]: curs.fetchall()
Out[73]: [(u'3', u'4', u'Charlie'), (u'5', u'6', u'Damion'), (u'78', u'90', u'Sidney')]

Be careful if your data involves Unicode, as the Python 2 csv module is known not to be Unicode-safe. (I recently had to write my own reader for that very reason).