事务回滚

Question

I have a big list which itself is consisted of 53,000,000 smaller lists as elements. And I want to submit each of these smaller lists as a row to a db in batches with the batch size of 1,000,000, meaning that every time the script connects to the db, it submits 1000,000 elements, then it disconnects from the db, and it connects again to submit another 1,000,000 rows.

Now my problem is that, if an error happens in the middle, for ex after submitting 50,000,000 rows, I need to delete all the rows in the db and try submitting everything from beginning.

I was thinking maybe I can use rollback(), to remove all 50,000,000 rows which has been added by now, but as long as I am using a loop, I do not know how I can rollback all 50,000,000 rows which are submitted in batches.

does any one have a suggestion?

here is my script: "results" is the list with 53,000,000 smaller lists as elements.

batch = []
counter = 0
BATCH_SIZE =1000000
cursor_count = 0

def prepare_names(names):
    return [w.replace("'", '') for w in names]

for i in range(len(results)):
    if counter < BATCH_SIZE:
        batch.append(prepare_names([results[i][0], results[i][1], results[i][2]]))  # batch => [[ACC1234.0, 'Some full taxa name'], ...]
        counter += 1
    else:
        batch.append(prepare_names([results[i][0], results[i][1], results[i][2]]))

        values = (", ").join([f"('{d[0]}', '{d[1]}', '{d[2]}')" for d in batch])
        sql = f"INSERT INTO accession_taxonomy(accession_number, taxon_id, taxonomy) VALUES {values}"

        try:
            cursor.execute(sql)
            db.commit()
        except Exception as exception:
            print(exception)
            print(f"Problem with query: {sql}")

        print(cursor.rowcount, "Records Inserted")
        cursor_count += cursor.rowcount
        counter = 0
        batch = []
else:
    if batch:
        values = (", ").join([f"('{d[0]}', '{d[1]}', '{d[2]}')" for d in batch])
        sql = f"INSERT INTO accession_taxonomy(accession_number, taxon_id, taxonomy) VALUES {values}"

        try:
            cursor.execute(sql)
            db.commit()
        except Exception as exception:
            print(exception)
            print(f"Problem with query: {sql}")

        print(cursor.rowcount, "Records Inserted")
        cursor_count += cursor.rowcount

print("Total Number Of %s Rows Has Been Added." %(cursor_count))
db.close()

Answer 1

There is no rollback after commit .

concider this:

1st Attempt 1M rows : committed
2nd Attempt 1M rows : committed
3rd Attempt 1m rows : error

You can only rollback the 3rd attempt. 1st and 2nd are done.

workaround modify your accession_taxonomy table and add a field something called insertHash . Your batch update process will have an unique value for this field - for this batch exectuion. let's say todaysDate - and if any of your insert steps fails you can then do

Delete T from accession_taxonomy T Where T.insertHash ='TheValueUSet'

so essentially it becomes like this:

1st Attempt 1M rows : committed
2nd Attempt 1M rows : committed
3rd Attempt 1m rows : error
Delete AllRows where insertHash = 'TheValueUSet'

Having said that , are you sure you want to shoot 1m rows? have you checked if your server is capable of accepting that large packet?

Answer 2

I would use some flags to make sure that

• something was inserted
• nothing wrong happened

And then, use those flags to choose to commit or to rollback, such as :

nothing_wrong_happened = True
something_was_inserted = False

for i in range(len(results)):

    # Your code that generates the query

        try:
            cursor.execute(sql)
            something_was_inserted = True  # <-- you inserted something
        except Exception as exception:
            nothing_wrong_happened = False # <-- Something bad happened
            print(exception)
            print(f"Problem with query: {sql}")

        # the rest of your code
else:

    # Your code that generates the query

        try:
            cursor.execute(sql)
            something_was_inserted = True  # <-- you inserted something
        except Exception as exception:
            nothing_wrong_happened = False # <-- Something bad happened
            print(exception)
            print(f"Problem with query: {sql}")

        # the rest of your code

# The loop is now over
if (something_was_inserted):
    if (nothing_wrong_happened):
        db.commit()   # commit everything
    else:
        db.rollback() # rollback everything