psycopg2-更快地插入具有多列的多行

Question

I'm trying to insert multiple rows into my database, and currently I do not know a way to insert them all at the same time or any other method which will help save time (sequentially it takes about ~30s for around 300 rows).

My 'rows' are are tuples in a list of tuples (converted into tuple of tuples), eg [(col0, col1, col2), (col0, col1, col2), (.., .., ..), ..]

def commit(self, tuple):
    cursor = self.conn.cursor()
    for tup in tuple:
        try:
            sql = """insert into "SSENSE_Output" ("productID", "brand", "categoryID", "productName", "price", "sizeInfo", "SKU", "URL", "dateInserted", "dateUpdated")
              values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""

            cursor.execute(sql, tup)
            self.conn.commit()
        except psycopg2.IntegrityError:
            self.conn.rollback()
            sql = 'insert into "SSENSE_Output" ' \
                  '("productID", "brand", "categoryID", "productName", "price", "sizeInfo", "SKU", "URL", "dateInserted", "dateUpdated")' \
                  'values (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s) on conflict ("productID") do update set "dateUpdated" = EXCLUDED."dateUpdated"'
            cursor.execute(sql, tup)
            self.conn.commit()
        except Exception as e:
            print(e)

I have also tried commiting after the for loop is done, but still results in the same amount of time. Are there any ways to make this insert significantly faster?

Answer 1

Building one large INSERT statement instead of many of them will considerably improve the execution time, you should take a look here . It is for mysql, but I think a similar approach apply for postgreSQL

Answer 2

In postgres you can use a format like:

INSERT INTO films (code, title, did, date_prod, kind) VALUES
('B6717', 'Tampopo', 110, '1985-02-10', 'Comedy'),
('HG120', 'The Dinner Game', 140, DEFAULT, 'Comedy');

Due to your record base exception handling you can better first resolve the duplicates before generating this query as the whole query might fail when an integrity error occurs.