upstream timed out error in Python Flask web app using Fabric on nginx

Question

I have my Python Flask web app hosted on nginx. While trying to execute a request it shows a timeout error in the nginx error log as shown below :

[error] 2084#0: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.2.224, server: 192.168.2.131, request: "POST /execute HTTP/1.1", upstream: "uwsgi://unix:/hom
e/jay/PythonFlaskApp/app.sock", host: "192.168.2.131:9000", referrer: " http://192.168.2.131:9000/ "

If I try to run the app locally it works fine and responds fine. Any one have any idea what might be wrong ?

the error found in browser console is :

Gateway Time-out

Here is the nginx config file:

server {
    listen 9000;
    server_name 192.168.2.131;

    location / {
        include uwsgi_params;
        proxy_read_timeout 300;
        uwsgi_pass unix:/home/jay/PythonFlaskApp/app.sock;
    }
}

And here is the Python Fabric code that i trying to execute. i'm not sure if this is causing the issue, but any waz here is the code :

from fabric.api import *

@application.route("/execute",methods=['POST'])
def execute():
    try:
        machineInfo = request.json['info']
        ip = machineInfo['ip']
        username = machineInfo['username']
        password = machineInfo['password']
        command = machineInfo['command']
        isRoot = machineInfo['isRoot']

        env.host_string = username + '@' + ip
        env.password = password
        resp = ''
        with settings(warn_only=True):
            if isRoot:
                resp = sudo(command)
            else:
                resp = run(command)

        return jsonify(status='OK',message=resp)
    except Exception, e:
        print 'Error is ' + str(e)
        return jsonify(status='ERROR',message=str(e))

I have a uWSGi config file for the web app and started it using an upstart script. Here is uwSGi conf file :

[uwsgi]
module = wsgi

master = true
processes = 5

socket = app.sock
chmod-socket = 660
vacuum = true

die-on-term = true

and here is upstart script

description "uWSGI server instance configured to serve Python Flask App"

start on runlevel [2345]
stop on runlevel [!2345]

setuid jay
setgid www-data

chdir /home/jay/PythonFlaskApp
exec uwsgi --ini app.ini

I have followed the below tutorial on running flask app on nginx

Answer 1

This is likely a problem with the Fabric task, not with Flask. Have you tried isolating / removing Fabric from the application, just for troubleshooting purposes? You could try stubbing out a value for resp , rather than actually executing the run / sudo commands in your function. I would bet that the app works just fine if you do that.

And so that would mean that you've got a problem with Fabric executing the command in question. First thing you should do is verify this by mocking up an example Fabfile on the production server using the info you're expecting in one of your requests, and then running it with fab -f <mock_fabfile.py> .

It's also worth noting that using with settings(warn_only=True): can result in suppression of error messages. I think that you should remove this, since you are in a troubleshooting scenario. From the docs on Managing Output :

warnings: Warning messages. These are often turned off when one expects a given operation to fail, such as when using grep to test existence of text in a file. If paired with setting env.warn_only to True, this can result in fully silent warnings when remote programs fail . As with aborts, this setting does not control actual warning behavior, only whether warning messages are printed or hidden.

As a third suggestion, you can get more info out of Fabric by using the show('debug') context manager , as well as enabling Paramiko's logging :

from fabric.api import env
from fabric.context_managers import show

# You can also enable Paramiko's logging like so:
import logging
logging.basicConfig(level=logging.DEBUG)

def my_task():
    with show('debug'):
        run('my command...')

The Fabric docs have some additional suggestions for troubleshooting: http://docs.fabfile.org/en/1.6/troubleshooting.html . (1.6 is an older/outdated version, but the concepts still apply.)