使用psycopg2在PostgreSQL中更新多行并记录错误

Question

I'm writing an application that connects to database and upserts multiple rows, it creates SAVEPOINT for every row, so I can rollback without breaking a transaction, if there is a mistake, and commits every 500 rows.

The problem is that it works extremely slow for remote database connections (postgresql database on DigitalOcean droplet) - it took about 35 minutes to process 1000 rows, when it was only 7 second with local database (which is also not quite fast, but ok).

I found post about upserting using one cursor.execute(), like here , but how should I catch errors if using this trick? Or what else should I do to make it work faster? Here is my code:

self.connection = psycopg2.connect(self.connection_settings)
self.cursor = self.connection.cursor()
for record in dbf_file:
    self.cursor.execute("SAVEPOINT savepoint;")
    try:
        self.send_record(record, where_to_save=database)
        self.count += 1
        self.batch_count += 1
        if self.batch_count >= BATCH_COUNT_MAX:
            self.connection.commit()
            self.cursor.close()
            self.cursor = self.connection.cursor()
            self.batch_count = 0
    except Exception:
        self.cursor.execute("ROLLBACK TO SAVEPOINT savepoint;")
        self.save_error(traceback.format_exc())
        self.error_count += 1
        self.batch_count += 1

        if self.batch_count == BATCH_COUNT_MAX:
            self.connection.commit()
            self.cursor.close()
            self.cursor = self.connection.cursor()
            self.batch_count = 0
    else:
        if self.batch_count != 0:

Answer 1

Given you are already using files, I'd suggest:

1. Make those files CSV (500 or what have you rows each)
2. Upload them to the server (scp/ftp/rsync)
3. Use psycopg's copy_from()

This way you will likely eliminate unneeded network overhead.