Serializing Sqlite3 in Python

Question

To fully utilize concurrency, SQLite3 allows threads to access the same connection in three ways:

1. Single-thread. In this mode, all mutexes are disabled and SQLite is unsafe to use in more than a single thread at once.
2. Multi-thread. In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads.
3. Serialized. In serialized mode, SQLite can be safely used by multiple threads with no restriction.

Does anyone know how I can make the connection serialized in Python.
Python has "check_same_thread" which allows switching between multi-threading and single-threading; however, I can not find out how I should make it serialized.

Answer 1

The Python SQLite module is not threadsafe. If you disable its checking then you need to ensure all code is serialized and that includes garbage collection. (My APSW module is threadsafe and also correctly handles the error message thread safety issues).

It is however safe to use multiple independent connections concurrently in the same process and I would recommend you do that. Additionally switch the database into write ahead logging mode and you should get very good performance even with lots of writing.

Answer 2

I wrote a library to solve this. Works for me.

https://github.com/palantir/sqlite3worker

Answer 3

The sqlite page http://www.sqlite.org/threadsafe.html says, "The default mode is serialized." Have you tested it and found this to not be true?

Edit:

---

If it fails to work, maybe ctypes? I have no idea if this would have any effect on the loaded sqlite module. I guess I sort of suspect it doesn't; as I'd imagine the sqlite3_initialize() function is likely called when the module is loaded? Or maybe only when you create a database object?

http://www.sqlite.org/c3ref/config.html

>>> import sqlite3
>>> import ctypes
>>> from ctypes.util import find_library
>>> sqlite_lib = ctypes.CDLL(find_library('sqlite3'))
>>> sqlite_lib.sqlite3_config(3) # http://www.sqlite.org/c3ref/c_abort.html
0   # no error....
>>> 

Answer 4

From Verse Quiz , you might be interested in the __init__ , __serve , and __fetch methods to get you started on creating a serialized SQLite3 database interface in Python. Hope that helps you further!

---

import _thread
import sqlite3
import queue

################################################################################

class Server:

    """Execute a protected SQLite3 database on a singular thread.

    Since a SQLite3 database can only accept queries on the thread that it
    was created on, this server receives requests through a queue and sends
    back the result through a list and mutex mechanism."""

    def __init__(self, *args):
        """Initialize the Server with a SQLite3 database thread."""
        self.__lock = _thread.allocate_lock()
        self.__lock.acquire()
        _thread.start_new_thread(self.__serve, args)
        self.__lock.acquire()
        del self.__lock
        if self.__error is not None:
            raise self.__error
        del self.__error

    def __serve(self, *args):
        """Run a server continuously to answer SQL queries.

        A SQLite3 connection is made in this thread with errors being raised
        again for the instantiator. If the connection was made successfully,
        then the server goes into a continuous loop, processing SQL queries."""
        try:
            database = sqlite3.connect(*args)
        except:
            self.__error = error = sys.exc_info()[1]
        else:
            self.__error = error = None
        self.__lock.release()
        if error is None:
            self.__QU = queue.Queue()
            while True:
                lock, one, sql, parameters, ret = self.__QU.get()
                try:
                    cursor = database.cursor()
                    cursor.execute(sql, parameters)
                    data = cursor.fetchone() if one else cursor.fetchall()
                    ret.extend([True, data])
                except:
                    ret.extend([False, sys.exc_info()[1]])
                lock.release()

    def fetch(self, one, sql, *parameters):
        """Execute the specified SQL query and return the results.

        This is a powerful shortcut method that is the closest connection
        other threads will have with the SQL server. The parameters for the
        query are dumped into a queue, and the answer is retrieved when it
        becomes available. This prevents SQLite3 from throwing exceptions."""
        lock, ret = _thread.allocate_lock(), []
        lock.acquire()
        self.__QU.put((lock, one, sql, parameters, ret))
        lock.acquire()
        if ret[0]:
            return ret[1]
        raise ret[1]