Data not being saved in Postgresql using Python

Question

I am trying to save information to my database in Postgresql using Python script. For a More General explanation I am Making a WebApp using Rails, in which I call a job to run a python script to process some images and data, after that, I wish to save them to my database.

Other things, I'm uploading an image to AWS S3 which I wait for the uploading to finish before saving the info into the database.

The python code I am using to upload the information is the following:

# Function which connects wit postgresql and uploads into the database
#....(Asume param and model_id are known variables, which i do not post it here.) 

def insert_param(param, model_id):
    con = None
    try:
        con = psycopg2.connect("host='localhost' 
                               dbname='app_development' 
                               user='usr' 
                               password=''")
        wait(con)
        cur = con.cursor()
        cur.execute("UPDATE M_Param SET Model=%s WHERE Id=%s",(param, model_id))
        con.commit()

    except Exception, e:
         print  e
         if con:
            con.rollback()

        print 'Error %s' % e
        sys.exit(1)

    finally:   
        if con:
            con.close()

#Function which wait for the connection to the database to be ready.
def wait(conn):
    while 1:
        state = conn.poll()
        if state == psycopg2.extensions.POLL_OK:
            print "Poll Conection OK"
            break
        elif state == psycopg2.extensions.POLL_WRITE:
            print "Poll writting"
            select.select([], [conn.fileno()], [])
        elif state == psycopg2.extensions.POLL_READ:
            print "Poll reading"
            select.select([conn.fileno()], [], [])
        else:
            raise psycopg2.OperationalError("poll() returned %s" % state)

#Here I call the first function
insert_count(param,model_id)

I am having problems sometimes uploading data to postgresql, random cases in which the information is not uploading. I tried fetching the data from the database afterwards it commits, but I got nothing, nor valuable information from those tries.

Here are my questions:

• Do I have something wrong in the way I am saving and committing the information to my database?
• Do I need to wait for the commit to return an "ok" state using some callback before proceed on running any more code?

Any help, reference or feedback is greatly welcome.

Answer 1

I found the problem, it is not in the python code, but in the way the python script is being executed .

When I call the python script, the interpreter is called, but before the python task is finished I already call the script another time, which makes the interpreter to close and reopen, not letting the previous task to finish (ergo, not committing to the database).

Some posible ways to avoid this problem are:

• Queueing in a list the different task to a always running python script in another server.
• Using sidekiq Gem to make all python scripts run in differents threads. (Or some other backend threadings methods)

I hope this findings helps others in the same position as me.