Flask application being run by gunicorn gets hanged after some time

Question

I am facing a unique problem from the past 30 days. After trying a hell lot of stuff, I am seeking support from the community.

I have made a deep learning based web application using python flask. Backend is written in python (deep learning code) and front end is being served using HTML, JS and Bootstrap. The deployment is done using Gunicorn. Also, the app is https enabled and deployed on GCP.

The application runs fine for some time, after that it hangs. Although the python process is still running, it stops serving and responding for any API requests. New hits do not reach the python code. This behavior is very random in terms of running time. Sometimes it takes 4-5 hours to stop and some other times it takes even 2 days to stop. Then I manually start the application using gunicorn command and it works.

Things I have tried -

• Checked system memory using htop when the application is in the hanging stage. But it seems fine.
• I have tried to test the APIs using Jmeter (hitting multiple requests in a loop in sequence) and the system does not hang up.
• I even ran the application using uwsgi + apache , but still, the problem persists.

Here is the command being used to run the gunicorn server -

gunicorn -b 0.0.0.0:443 --threads=4 --certfile=path_to_certificate_file --keyfile=path_to_key_file server:app --max-requests 1000 --access-logfile /var/log/gunicorn/gunicorn-access.log --error-logfile /var/log/gunicorn/gunicorn-error.log --capture-output --log-level debug --logger-class=simple --daemon

I am still not able to diagnose the exact problem and replicate it. Looking for any specific direction to explore. Feel free to share your hypothesis/thoughts. Let me know if any other information is required from my side in order to make it intuitive further.

Answer 1

I've also been experiencing this issue with a gunicorn/flask app:

Requests load only within the first few minutes after gunicorn starts. When left alone for a few minutes, gunicorn stops responding to requests (browser perpetually displays "loading" icon animation). Gunicorn error log shows nothing.

My current workaround is to create an arbitrary "keep alive" call to the flask app from the client html file.

So in the Flask app file:

@app.route('/keep_alive/<val>')
def keep_alive(val):
    return json.dumps({'success':val}) 

And in the html's JS block:

function keepAlive() {
        var t = setTimeout(keepAlive, 60000);
        var d = new Date();
        var n = d.getTime();
        $.getJSON(base_url + '/keep_alive/' + n.toString(), function (data) {
                console.log('keep alive return ' + data)
            });
    } 
keepAlive();

For what it's worth, the gunicorn setting --keep-alive=21600 didn't have any noticeable effect. It would be great if someone here could enlighten me.

Thanks