在sqlalchemy中并行运行查询

Question

I have a table (say table_A) in a db A, there are two other tables in db's B and C. I am running query like :

session.query(Table_A).filter(Table_A.date >= time1, InventoryLog.audit_date < time2)

I am running it in a loop, adding the rows in table_B and updating time1,time2 in each iteration.

Now, I want to run another query in parallel on the same (or different table) for different time intervals and insert in another table_C (loop). The important point is to run them parallely.

How should I go about it in sqlalchemy? Should I use celery for this or will that be an overkill?

Thanks in advance!

Answer 1

You can use threading for paralel processing.

from threading import Thread

def fn1(arg1):
    while True:
        session.query(Table_A).filter(Table_A.date >= time1, InventoryLog.audit_date < time2)
        # do other stuff


def fn2(arg1):
    while True:
        session.query(Table_B).filter(Table_B.date >= time1, InventoryLog.audit_date < time2)
        # do other stuff


def fn3(arg1):
    while True:
        session.query(Table_C).filter(Table_C.date >= time1, InventoryLog.audit_date < time2)
        # do other stuff


t1 = Thread(target=fn1, args=(1,))
t2 = Thread(target=fn2, args=(1,))
t3 = Thread(target=fn3, args=(1,))

t1.start()
t2.start()
t3.start()