Gunicorn RecursionError 与 gevent 和 python 3.6.2 中的请求

Question

I have a simple flask script that uses requests to make http requests to third party web service. The way I run the script in gunicorn is

gunicorn abc:APP -b 0.0.0.0:8080 -w 4 -k gevent --timeout 30 --preload

However, after I upgrade the code to python 3.6.2, I can still run the server, but whenever the webserver received a request, it shows

RecursionError: maximum recursion depth exceeded while calling a Python object

on every worker, and the server seems are still running. When I change the running command to

gunicorn abc:APP -b 0.0.0.0:8080 -w 4 --timeout 30 --preload

It all works again. So is there any issue with gunicorn's async worker and requests in python 3.6.2? Is there a way to fix this?

(This question is also asked at https://github.com/benoitc/gunicorn/issues/1559 )

Answer 1

This is because of the ssl is imported before monkey patch, gunicorn import the ssl module(config.py in gevent) when loading config，however the monkey patch is called when init the worker or at the top of app.py file which is definitely after import of ssl, this is why we get the warning.

A simple solution is to use a config file for gunicorn. we could do gevent monkey patch at the beginning of config file, and start gunicorn with the config file, in this way, the monkey patch could be completed before import ssl, and thus avoid this problem.

A config file named gunicorn_config.py could contain lines below:

import gevent.monkey
gevent.monkey.patch_all()

workers = 8

and then we could start gunicorn with

gunicorn --config config.py --worker-class gevent --preload -b 0.0.0.0:5000 app:app

More information could be found here

Answer 2

Please see https://github.com/benoitc/gunicorn/issues/1559 . This may possibly be fixed in the next version of gunicorn, but unfortunately you may have to stay with python 3.5.2 if you don't want to break gunicorn.