Flask Python Gunicorn MySql - MySQL Connection not available

Question

I read so many question about this error, but none of them fix my issue.

I am not really familiar with Python server side of things, and I am getting this error on an api that runs with Gunicorn and Flask.

Tried to: - Remove all cursor.close() - Add cursor.close() right after the api call before the return

Here is my connector:

db = mysql.connector.connect(
    host="127.0.0.1",
    user="root",
    port=3306,
    password='password',
    database='db'
)

An example of api call:

@api.route('/update-entities', methods=['POST'])
def update_entities():
    try:
        validation = call_controller(ApiController(request.json).update_entities, 'update_entities')
        cursor = c.pearch_db.cursor()
        cursor.close()
        return jsonify({'status': validation.status, 'status_code': validation.status_code, 'message': validation.message}), validation.status_code
    except Exception:
        cursor = c.pearch_db.cursor()
        cursor.close()
        return jsonify(traceback.format_exc()), 500

Error:

Dec 13 11:47:43 cybertruck gunicorn[35123]:     return self.wsgi_app(environ, start_response)
Dec 13 11:47:43 cybertruck gunicorn[35123]:   File "/var/www/russell-python/Api/middleware.py", line 17, in __c
Dec 13 11:47:43 cybertruck gunicorn[35123]:     organization = Organization.find_one_by_client_id_and_client_se
Dec 13 11:47:43 cybertruck gunicorn[35123]:   File "/var/www/russell-python/Common/Model/organization.py", line
Dec 13 11:47:43 cybertruck gunicorn[35123]:     results = h.query(sql)
Dec 13 11:47:43 cybertruck gunicorn[35123]:   File "/var/www/russell-python/Common/helpers.py", line 19, in que
Dec 13 11:47:43 cybertruck gunicorn[35123]:     cursor = db.cursor(dictionary=dictionary)
Dec 13 11:47:43 cybertruck gunicorn[35123]:   File "/var/www/russell-python/pearch/local/lib/python3.6/site-pac
Dec 13 11:47:43 cybertruck gunicorn[35123]:     raise errors.OperationalError("MySQL Connection not available."
Dec 13 11:47:43 cybertruck gunicorn[35123]: mysql.connector.errors.OperationalError: MySQL Connection not avail

So this error forces my restart the service that takes care of gunicorn every time otherwise the api will not work. Restarting the service resolve the issue for another few minutes. The api is used by 5 people at the time maybe.

Here is the service configuration in ubuntu server (the restart on failure doesnt workaround the issue unfortunately)

[Unit]
Description=Gunicorn instance to serve Pearch Python API
After=network.target
[Service]
Restart=on-failure
RestartSec=5s
#We will give our regular user account ownership of the process since it owns all of the relevant files
User=www-data
#give group ownership to the www-data group so that Nginx can communicate easily with the Gunicorn processes.
Group=www-data
# We'll then map out the working directory and set the PATH environmental variable so that the init system knows where our the executables for the process are located (within our virtual environment).
WorkingDirectory=/var/www/russell-python
Environment="PATH=/var/www/russell-python/pearch/bin"
# We'll then specify the commanded to start the service
ExecStart=/var/www/russell-python/pearch/bin/gunicorn  --timeout 300 --workers 25 --bind unix:app.sock -m 007 wsgi:api
# This will tell systemd what to link this service to if we enable it to start at boot. We want this service to start when the regular multi-user system is up and running:
[Install]
WantedBy=multi-user.target 

Mysql has this configuration:

max_connections         = 1000
connect_timeout         = 5
wait_timeout            = 600
max_allowed_packet      = 16M
thread_cache_size       = 128
sort_buffer_size        = 4M
bulk_insert_buffer_size = 16M
tmp_table_size          = 32M
max_heap_table_size     = 32M

And nginx site is like this:

server {
    server_name serverher;
    proxy_connect_timeout 75s;
    proxy_read_timeout 300s;

    location / {
        fastcgi_buffers 8 16k;
        fastcgi_buffer_size 32k;
        fastcgi_connect_timeout 300;
        fastcgi_send_timeout 300;
        fastcgi_read_timeout 300;
        proxy_connect_timeout 75s;
        proxy_read_timeout 300s;
        include proxy_params;
        proxy_pass http://unix:/var/www/russell-python/app.sock;
    }

    access_log /var/log/nginx/russell-python-access.log;
    error_log /var/log/nginx/russell-python-error.log;
}

I believe that mysql and nginx are all fine, also gunicorn which has a huge timeout setting, I don't know what else could be checked.

Answer 1

For anyone looking at this I had the same issue and traced it back to another bug as the root cause.

I was running out of connections in my pool of 32 so after some research i identified that instead of using cnx.close() I should reset_session followed by add_connection to allow the 32 connections to stay as 32 in the pool and not decrease over time (effected batch jobs and API calls making more than 32 calls per request).

The issue that was the cause of "MySQL Connection not available" was with the use of reset_session "MySQL Connection not available" if the connection .is_connected() # False occurs.

To fix both bugs I continued to use my own close as the following;

    def close(self):
        try:
            cnx = self.con
            if self.con.reset_session:
                cnx.reset_session()
        except OperationalError as err:
            if err.msg != 'MySQL Connection not available.':
                # This close method only needs to close available connections
                raise err
        finally:
            self.con.add_connection(cnx)
            self.con = None

So If you ever see "MySQL Connection not available" go and find out if you are using reset_session and handle the bug there.

Hope this helps