Python sqlite3 和并发

Question

I have a Python program that uses the "threading" module. Once every second, my program starts a new thread that fetches some data from the web, and stores this data to my hard drive. I would like to use sqlite3 to store these results, but I can't get it to work. The issue seems to be about the following line:

conn = sqlite3.connect("mydatabase.db")

• If I put this line of code inside each thread, I get an OperationalError telling me that the database file is locked. I guess this means that another thread has mydatabase.db open through a sqlite3 connection and has locked it.
• If I put this line of code in the main program and pass the connection object (conn) to each thread, I get a ProgrammingError, saying that SQLite objects created in a thread can only be used in that same thread.

Previously I was storing all my results in CSV files, and did not have any of these file-locking issues. Hopefully this will be possible with sqlite. Any ideas?

Answer 1

Contrary to popular belief, newer versions of sqlite3 do support access from multiple threads.

This can be enabled via optional keyword argument check_same_thread :

sqlite.connect(":memory:", check_same_thread=False)

Answer 2

You can use consumer-producer pattern. For example you can create queue that is shared between threads. First thread that fetches data from the web enqueues this data in the shared queue. Another thread that owns database connection dequeues data from the queue and passes it to the database.

Answer 3

The following found on mail.python.org.pipermail.1239789<\/a>

I have found the solution. I don't know why python documentation has not a single word about this option. So we have to add a new keyword argument to connection function and we will be able to create cursors out of it in different thread. So use:

sqlite.connect(":memory:", check_same_thread = False)

Answer 4

Switch to multiprocessing<\/a> . It is much better, scales well, can go beyond the use of multiple cores by using multiple CPUs, and the interface is the same as using python threading module.

Answer 5

You shouldn't be using threads at all for this. This is a trivial task for twisted<\/a> and that would likely take you significantly further anyway.

Answer 6

Or if you are lazy, like me, you can use SQLAlchemy<\/a> . It will handle the threading for you, ( using thread local, and some connection pooling<\/a> ) and the way it does it is even configurable<\/a> .

Answer 7

您需要在对数据库的每个事务<\/em>之后使用session.close()<\/code>以便在同一个线程中使用相同的游标，而不是在导致此错误的多线程中使用相同的游标。

"

Answer 8

使用threading.Lock()

Answer 9

I could not find any benchmarks in any of the above answers so I wrote a test to benchmark everything.

I tried 3 approaches

1. Reading and writing sequentially from the SQLite database
2. Using a ThreadPoolExecutor to read/write
3. Using a ProcessPoolExecutor to read/write

The results and takeaways from the benchmark are as follows

1. Sequential reads/sequential writes work the best
2. If you must process in parallel, use the ProcessPoolExecutor to read in parallel
3. Do not perform any writes either using the ThreadPoolExecutor or using the ProcessPoolExecutor as you will run into database locked errors and you will have to retry inserting the chunk again

You can find the code and complete solution for the benchmarks in my SO answer HERE Hope that helps!

Answer 10

I would take a look at the y_serial Python module for data persistence: http://yserial.sourceforge.net

which handles deadlock issues surrounding a single SQLite database. If demand on concurrency gets heavy one can easily set up the class Farm of many databases to diffuse the load over stochastic time.

Hope this helps your project... it should be simple enough to implement in 10 minutes.

Answer 11

I like Evgeny's answer - Queues are generally the best way to implement inter-thread communication. For completeness, here are some other options:

Answer 12

You need to design the concurrency for your program. SQLite has clear limitations and you need to obey them, see the FAQ<\/a> (also the following question).

"

Answer 13

Scrapy<\/a> seems like a potential answer to my question. Its home page describes my exact task. (Though I'm not sure how stable the code is yet.)

"

Answer 14

Please consider checking the value of THREADSAFE for the pragma_compile_options of your SQLite installation. For instance, with

SELECT * FROM pragma_compile_options;

If THREADSAFE is equal to 1, then your SQLite installation is threadsafe, and all you gotta do to avoid the threading exception is to create the Python connection with checksamethread equal to False . In your case, it means

conn = sqlite3.connect("mydatabase.db", checksamethread=False)

That's explained in some detail in Python, SQLite, and thread safety

Answer 15

The most likely reason you get errors with locked databases is that you must issue

conn.commit()