How can I track time a python subprocess while it's running dynamically and update a database?

Question

I have a subprocess that encodes a video , and what I would love to do is update at database record with the time it is taking to encode the video (so I can print it out in ajax on a web page)

I am very close - this code I have so far updates the database and encodes the video - but the process/loop gets stuck on the final db.commit() and never exits the while True: loop. Is there a better way to do this? Here is the code I am tinkering with:

time_start = time.time()

try:
        p = subprocess.Popen(["avconv" , "-y" , "-t" , "-i" , images , "-i" , music_file  , video_filename],  universal_newlines=True, stdout=subprocess.PIPE)

    while True:
            time_now = time.time()
            elapsed_time = time_now - time_start
            progress_time = "ENCODING TIME" + str(int(elapsed_time)) + " Seconds "
            cursor.execute("UPDATE video SET status = %s WHERE id = %s" ,[progress_time , video_id] )
            db.commit()
    out, err = p.communicate()
    retcode = p.wait()

except IOError:
    pass
else:
    print "] ENCODING OF VIDEO FINISHED:" + str(retcode)

Answer 1

You're right, because you have no way of exiting your infinite loop, it will just spin forever. What you need to do it call check p.poll() to see if the process has exited (it will return none if it hasn't). So, something like:

while True:
    if p.poll():
         break
    ... other stuff...

or better yet:

while p.poll() == None:
    .... other stuff....

will cause your loop to terminate when the subprocess is complete. then you can call p.communicate() to get the output.

I would also suggest using a sleep or delay in there so that your loop doesn't spin using 100% of your CPU. Only check and update your database every second, not continuously. So:

while p.poll() == None:
    time.sleep(1)
    ...other stuff...

Answer 2

In addition to the infinite loop issue pointed out by @clemej , there is also a possibility of a deadlock because you don't read from p.stdout pipe in the loop despite stdout=subprocess.PIPE ie, while p.poll() is None: will also loop forever if avconv generates enough output to fill its stdout OS pipe buffer.

Also, I don't see the point to update the progress time in the database while the process is still running. All you need is two records:

video_id, start_time # running jobs
video_id, end_time   # finished jobs

If the job is not finished then the progress time is current_server_time - start_time .

If you don't need the output then you could redirect it to devnull:

import os
from subprocess import call
try:
    from subprocess import DEVNULL # Python 3
except ImportError:
    DEVNULL = open(os.devnull, 'r+b', 0)

start_time = utcnow()
try:
    returncode = call(["avconv", "-y", "-t", "-i", images,
                       "-i", music_file, video_filename],
                       stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL)
finally:
    end_time = utcnow()