Flask, Gunicorn, Nginx :: IOError: [Errno 32] Broken pipe

Question

I'm trying to move a project from development to production. (At dev stage, I was using only Flask , and now I'm running it behind Gunicorn with Nginx .)

I'm having a problem to serve a specific page, songs.html .

The page loads correctly with a dummy variable ( jukebox = [whatever] ), but in real life I'm using a generator, like so:

playlist= query_playlist(p)
jukebox = next(playlist)

return render_template('songs.html',
                        jukebox=jukebox)

and this function takes a while (say 2s) to return results...but results are not being served, and process just hangs after results are returned.

I run the app like so:

(appenv)$gunicorn -c gconfig.py app:app

wsgi.ppy

from app import app as application

if __name__ == "__main__":
    application.run(host='0.0.0.0')

gconfig.py:

workers = 16
worker_class = 'sync'
worker_connections = 1000
timeout = 120 # changed this from 30 up
keepalive = 2

nginx.conf ( brew installation)

server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
        proxy_pass         http://127.0.0.1:8000/;
        proxy_redirect     off;

        proxy_set_header   Host                 $host;
        proxy_set_header   X-Real-IP            $remote_addr;
        proxy_set_header   X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto    $scheme;
        }

If I run the app using python app.py , back to dev stage:

if __name__ == '__main__':
    app.run(use_reloader=True, threaded=True, debug=True)

at least I get the following traceback:

Error on request:
Traceback (most recent call last):
  File "/Users/me/Documents/Code/Apps/app/appenv/lib/python2.7/site-packages/werkzeug/serving.py", line 270, in run_wsgi
    execute(self.server.app)
  File "/Users/me/Documents/Code/Apps/app/appenv/lib/python2.7/site-packages/werkzeug/serving.py", line 261, in execute
    write(data)
  File "/Users/me/Documents/Code/Apps/app/appenv/lib/python2.7/site-packages/werkzeug/serving.py", line 236, in write
    self.send_header('Server', self.version_string())
  File "/Users/me/anaconda2/lib/python2.7/BaseHTTPServer.py", line 412, in send_header
    self.wfile.write("%s: %s\r\n" % (keyword, value))
IOError: [Errno 32] Broken pipe

so does anyone know how to fix this?

EDIT

I get the same traceback even with results taking 2s to be processed.

Answer 1

According to Flask docs (and following #arielnmz sugestion), you can stream your content:

from flask import Response, stream_with_context

@app.route('/playlist')
def generate_playlist():
    def generate():
        jukebox = query_playlist(p)
        for i in jukebox:
            yield i[0]['artist'] 
    return Response(stream_with_context(generate()))

Each yield expression is directly sent to the browser.

The trick is to have an inner function that uses a generator to generate data and to then invoke that function and pass it to a response object.

This works.