使用数据库的多线程程序的架构

Question

I've got a fairly simple Python program as outlined below:

It has 2 threads plus the main thread. One of the threads collects some data and puts it on a Queue.

The second thread takes stuff off the queue and logs it. Right now it's just printing out the stuff from the queue, but I'm working on adding it to a local MySQL database.

This is a process that needs to run for a long time (at least a few months).

How should I deal with the database connection? Create it in main, then pass it to the logging thread, or create it directly in the logging thread? And how do I handle unexpected situations with the DB connection (interrupted, MySQL server crashes, etc) in a robust manner?

Answer 1

How should I deal with the database connection? Create it in main, then pass it to the logging thread, or create it directly in the logging thread?

I would perhaps configure your logging component with the class that creates the connection and let your logging component request it. This is called dependency injection , and makes life easier in terms of testing eg you can mock this out later.

If the logging component created the connections itself, then testing the logging component in a standalone fashion would be difficult. By injecting a component that handles these, you can make a mock that returns dummies upon request, or one that provides connection pooling (and so on).

How you handle database issues robustly depends upon what you want to happen. Firstly make your database interactions transactional (and consequently atomic). Now, do you want your logger component to bring your system to a halt whilst it retries a write. Do you want it to buffer writes up and try out-of-band (ie on another thread) ? Is it mission critical to write this or can you afford to lose data (eg abandon a bad write). I've not provided any specific answers here, since there are so many options depending upon your requirements. The above details a few possible options.