Insert Python list of dictionaries into PSQL database

Question

I have a list of dictionaries with the same keys (built in Python) which I would like to insert into a PSQL database. Key names coincide with the columns of the PSQL database. Could someone please suggest something to do it efficiently?

Answer 1

You can use .executemany method. Example using psycopg2 :

cursor.executemany(
    'INSERT INTO mytable (field_a, field_b, filed_c) '
    'VALUES (%(field_a)s, %(field_b)s, %(field_c)s)',
    data
)

data can be your list of dicts.

Answer 2

You can use executemany in PSQL to insert multiple records at once as follows:

conn=psycopg2.connect("dbname='db_name' user='db_user' password='db_pass'")

data = [{"col1":"data11", "col2":"data21"},
            {"col1":"data12", "col2":"data22"},
            {"col1":"data13", "col2":"data23"}]


cur = conn.cursor()
cur.executemany("""INSERT INTO bar(col1,col2) VALUES (%(col1)s, %(col2)s)""", data)

If you do not have dict structure, you need to make sure that data is in correct sequences of your columns in your table as :

cur.executemany(
    """INSERT INTO bar(col1,col2) 
       VALUES (%s,%s)""", data)

and data should be format :

data = [['data11', 'data21'], ['data12', 'data22']]

Is that what you are trying to do? Let me know, it works.

Answer 3

From the docs :

The current implementation of executemany() is (using an extremely charitable understatement) not particularly performing.

As per this thread, You should be using cursor.copy_from or extras.execute_values which perform single, bulk inserts.

copy_from

data = [{'col1': 1, 'col2': 2}, {'col1': 3, 'col2': 4}, {'col1': 5, 'col2':6}]
f = IteratorFile(("{0}\t{1}".format(k['col1'], k['col2']) for k in data.keys()))
cursor.copy_from(f, 'table_name', columns=('col1', 'col2'))

Please see here for a benchmark test compared to cursor.execute .

execute_values

data = [{'col1': 1, 'col2': 2}, {'col1': 3, 'col2': 4}, {'col1': 5, 'col2':6}]
insert_query = 'insert into table_name (a, b) values %s'
psycopg2.extras.execute_values(
    cursor, insert_query, [tuple(d) for d in data]
)