Update SQlite3 database with other SQlite3 database in Python

Question

I have two SQlite databases "archive_db" and "recent_db" with the same tables. I want to update the content of "archive_db" with the content of "recent_db". I tried:

dbconn_archive = sqlite3.connect(archive_db)
dbconn_recent = sqlite3.connect(recent_db)
query = "".join(line for line in dbconn_recent.iterdump())
dbconn_archive.executescript(query)

but this does not work as the tables already exist in archive_db.

So what is the correct query to update "archive_db" with "recent_db" ?

Answer 1

The method you used is good for creating a new database from an existing database from scratch. The best way I can think of for updating you database will actually require some more code.

1. You will have to determine what tables are in recent_db .
2. Iterate through the list of available tables in recent_db :
   • Append the records from recent_db to archive_db
   • Assuming you have enforced some type of uniqueness for your columns in archive_db , you can catch SQLite's IntegrityError and just skip the records that already exist in the table based on the uniqueness condition

Note : I'm going to assume that all the tables in recent_db also exist in archive_db . If not, then you will have to do an additional check to create the new tables in archive_db , but that would be out of the scope of this problem. I'm also going to assume that you don't really want to update the sqlite_sequence table since it's an internal table used to track ROWIDs. And lastly, I'm going to assume that the order of columns in your recent_db tables are the same for archive_db

import sqlite3

recent_con = sqlite3.connect('recent.db')
archive_con = sqlite3.connect('archive.db')
recent_cur = recent_con.cursor()
archive_cur = archive_con.cursor()

# table query
tbl_q = """
SELECT name
FROM sqlite_master
WHERE type='table'
    AND name != 'sqlite_sequence'
"""

tables = [t[0] for t in recent_cur.execute(tbl_q)]

for table in tables:
    for row in recent_cur.execute("SELECT * FROM {}".format(table)):
        try:
            archive_cur.execute("INSERT INTO {} VALUES ({})".format(table, ','.join('?' * len(row))), row)
        except sqlite3.IntegrityError:
            pass  # skip record since unique constraint failed
    archive_con.commit()  # commit inserts

archive_cur.close()
recent_cur.close()
archive_conn.close()
recent_conn.close()