Non-blocking Python database call

Question

I need to make many concurrent database calls while allowing the program to continue to run. When the call returns, it sets a value.

If the queries were known right away, we can use a ThreadPoolExecutor for example. What if we don't have all queries ready ahead of time but we are running them as we go? For example, we are traversing a linked list and at each node we want to make a database query and set a value based on the response.

The task here is to not wait until the database result is returned before proceeding to the next node.

Is it possible? One idea would be to create a Thread object. Maybe we can use asyncio to our advantage. The advantage of traversing and requesting as we go over traversing, collecting all the nodes and running them all at once is that the database won't be overwhelmed as much however the difference might be minimal.

Thanks!

Answer 1

If you're using SQLite then you can use https://pypi.org/project/sqlite3worker/

If not, you can use the Queue library. You can queue the items from your thread calls.

And a condition to execute the queue item sequentially.

You can check the implementation of sqlite3worker and implement similarly for your own database.

PS: Databases like SQL Server allow you to make consequent calls by default, you needn't worry about being threadsafe.