Connection.commit() does not persist data in Django Test

Question

I am attempting to test if my db_insert() method works.

def db_insert(data, insert_query, limit=100):
    """Persist data to database
    :param data: Tuple of tuples
    :param insert_query: String
    :param limit: Integer
    """
    # Insert / Merge
    cursor = conn.cursor()
    try:
        for i in range(0, int(len(data) / limit) + 1):
            sliced_data = data[i * limit:i * limit + limit]
            if sliced_data:
                cursor.executemany(insert_query, sliced_data)
                conn.commit()
    except Exception as e:
        conn.rollback()
        raise DBException('ScriptName:db_manager.py,ErrorType:{}Impact:Unable to insert into database,'
                          'ErrorMessage:{}'.format(type(e).__name__, e))

This method is called by another method:

def insert(cls, new_list):
    """Insert new data to DB
    :param new_list: List
    """
    try:
        insert_query = "insert into TABLE {} values {}" \
            .format(tuple(cls.TABLE_ATTR[1:]), ('%s',) * len(cls.TABLE_ATTR[1:]))
        insert_query = insert_query.replace('\'', '')
        db_insert(new_list, insert_query)
    except Exception as e:
        logger.exception(e)

Lastly, the insert() method is invoked by a test in a TestCase subclass:

def test_insert_method_success(self):
        SomeModule.insert(['text1', 'text2', 'text3', 1, settings.SCRIPT_RUN_TIME])
        cursor = conn.cursor()
        cursor.execute("select * from TABLE")
        data = cursor.fetchall()
        print(data)  # [1]

The output in [1] does return 2 tuples. These tuples however are data that have been added into the DB using models.save() in the setUp() .

Can someone show me why conn.commit() is not persisting the data into the database as expected in a real run?

Answer 1

Your test is failing because it's running inside a Django TestCase . That class uses transactions as a mechanism to run and then rollback tests. Since every test is run inside a transaction, any attempt to manually manage transactions within the test (such as your conn.commit() and conn.rollback() ) will wreak havoc.

For tests like that that you should instead use TransactionTestCase . That uses table truncation to undo database effects; it's slower, but won't interfere with your ability to manage transactions.

See the documentation for more detail:

Django's TestCase class is a more commonly used subclass of TransactionTestCase that makes use of database transaction facilities to speed up the process of resetting the database to a known state at the beginning of each test. A consequence of this, however, is that some database behaviors cannot be tested within a Django TestCase class....

A TransactionTestCase [unlike a TestCase ] may call commit and rollback and observe the effects of these calls on the database.