并发的sqlite用python写

Question

I've got a python application (Gtk) which uses threads to fetch information from certain sites and writes them to the datebase.

I've got a thread that checks for new updates at site1 , if there are updates I receive a json object ( json1 ). I will then iterate through json1 and insert the new information to the datebase, within json1 there is a result I need to use to fetch more information at site2 . I will recive a json object( json2 ) at site2 as well.

So the situation is something like this

def get_more_info(name):
    json2 = get(www.site2.com?=name....)
    etc

for information in json1:
    db.insert(information)
    get_more_info(information.name)

From this situation I see that there are a couple of ways of doing this.

get_more_info to return json object so that

for information in json1:
    db.insert(information)
    json2 = get_more_info(information.name)
    for info in json2:
        db.insert(info)
db.commit()

get_more_info to do the inserting

for information in json1:
    db.insert(information)
    get_more_info(information.name)
db.commit()

Both of these ways seem a bit slow since the main for loop will have to wait for get_more_info to complete before carrying on and both json1 and json2 could be large, there is also the possiblity that site2 is unavailiable at that moment, causing the whole transaction to fail. The application can still function without json2 , that data can be fetched at a later time if needed.

So I was thinking of passing information.name to a queue so that the main loop can continue and kick off a thread that will monitor that queue and excute get_more_info . Is this the right approach to take?

I know that sqlite does not perform concurrent writes, If I recall correctly if get_more_info tries to write while the main for loop is busy, sqlite will output OperationalError: database is locked .

Now what happends to get_more_info at that point, does it get put into sometype of write queue or does it wait for the main loop to complete and what happens to the main for loop when get_more_info is busying writing?

Will there be a need to go to another database engine?

Answer 1

Since you are using threads always, you can use an other thread to write to the database. In order to feed it with the data you should use a globally accessible Queue.Queue() ( queue.Queue() in Python3) instance. Using the instances get() method with block=true will make the thread wait for data to write.