Python on the web: executing code as it's processed?

Question

I made a python application that I'd like to deploy to the web. I'm on a Mac, so I enabled the web server and dropped it in my cgi-bin, and it works fine. The problem is, the application does some intensive computations, and I would really like to let the user know what's going on while it's executing.

Even though i have print statement scattered throughout the code, it doesn't output anything to my browser until the entire thing is done executing. Is there any way I can fix this to execute code as it's processed?

Answer 1

Instead of 'print', you might want to try

sys.stdout.write('something something something')
sys.stdout.flush()

That'll ensure that the web server isn't waiting for a buffer to fill up.

Answer 2

If sys.stdout.flush() didn't do the trick, the problem is likely to be resolved by chunked-encoding transfer.

To give a little bit of background, chunked-encoding defines a mechanism where up-front the server will tell the client 'My data stream has no limit', and as an efficiency the data is transferred in chunks as opposed to just streaming content willy-nilly.

Here's a simple example, the important is how you send the data and the headers you use.

Another aspect of this is what the browser actually does with the data as it comes in, even if your cgi is sending data to the browser it might just sit on it until it's done.

With the following example, curl shows each 'chunk' being downloaded correctly in a stream, Safari still hangs waiting for the CGI to complete.

#!/usr/bin/python
import time
import sys

def chunk(msg=""):
    return "\r\n%X\r\n%s" % ( len( msg ) , msg )

sys.stdout.write("Transfer-Encoding: chunked\r\n")
sys.stdout.write("Content-Type: text/html\r\n")

for i in range(0,1000):
    time.sleep(.1)
    sys.stdout.write( chunk( "%s\n" % ( 'a' * 80 ) ) )
    sys.stdout.flush()

sys.stdout.write(chunk() + '\r\n')

So if you just connect to this CGI with your browser, yeah, you won't see any changes - however if you use AJAX techniques and setup a handler every time you get data you'll be able to 'stream' it as it comes in.

Answer 3

Probably the best approach to something like this to seperate your concerns. Make an ajax-drive "console" type display, that for instance will poll a log file, which is written to in the worker process.