Python multiprocessing. Finish all processes at the same time

Question

My goal is upload some data to database. I'm using psycopg2 and it has a rule: all processes must have own database connection. In my case it means that I must commit in worker. The problem is that I can commit only all processes finish sql insert command. What I need:

from multiprocessing import Process

def worker1(s):    
    conn = psycopg2.connect("dbname=mydb user=postgres")
    cursor = conn.cursor()

    pg_cursor.execute(
            """ insert into "MyTable1"("Column1")
                values(%s)""", [1])

    #wait all processes
    conn.commit()       

def worker2(s):    
    conn = psycopg2.connect("dbname=mydb user=postgres")
    cursor = conn.cursor()

    pg_cursor.execute(
            """ insert into "MyTable2"("Column1")
                values(%s)""", [1])

    #wait all processes
    conn.commit()

if __name__ == '__main__':
    p1 = Process(target=worker1)
    p2 = Process(target=worker2)
    p1.start()
    p2.start()

How can I make all process wait untill finish sql commands? What is correct way to do this? This sql insert only for example, in real task i need to insert millions of records.

Answer 1

You can use a pair of multiprocessing.Event objects to allow both workers to tell the other that they're done, as well as to force them to wait for the other to signal back:

from multiprocessing import Process, Event

def worker1(s, my_e, other_e):    
    conn = psycopg2.connect("dbname=mydb user=postgres")
    cursor = conn.cursor()

    pg_cursor.execute(
            """ insert into "MyTable1"("Column1")
                values(%s)""", [1])

    #wait all processes
    my_e.set()
    other_e.wait()
    conn.commit()       

def worker2(s, my_e, other_e):    
    conn = psycopg2.connect("dbname=mydb user=postgres")
    cursor = conn.cursor()

    pg_cursor.execute(
            """ insert into "MyTable2"("Column1")
                values(%s)""", [1])

    #wait all processes
    my_e.set()
    other_e.wait()
    conn.commit()

if __name__ == '__main__':
    e1 = Event()
    e2 = Event()
    p1 = Process(target=worker1, args=(e1, e2))
    p2 = Process(target=worker2, args=(e2, e1))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

Just be careful about the possibility of a deadlock if one of the workers fails for some reason. You may want to set the optional timeout keyword argument of Event.wait to some fairly high value, and rollback if it expires. That or use a try / except in each worker that guarantees my_e gets set() , but also share a variable between the two workers that can be used to tell the other worker if a failure occurred.