为普罗米修斯 django 导出器重新/使用 redis 连接的最佳方法

Question

I am getting an error

redis.exceptions.ConnectionError: Error 24 connecting to redis-service:6379. Too many open files.
...
OSError: [Errno 24] Too many open files

I know this can be fixed by increasing the ulimit but I don't think that's the issue here and also this is a service running on a container. The application starts up correctly works for 48 hours correctly and then I get the above error. Which implies that the connections are growing over time exponentially.

What my application is basically doing

• background_task (ran using celery) -> collects data from postgres and sets it on redis
• prometheus reaches the app at '/metrics' which is a django view -> collects data from redis and serves the data using django prometheus exporter

The code looks something like this

views.py

from prometheus_client.core import GaugeMetricFamily, REGISTRY
from my_awesome_app.taskbroker.celery import app


class SomeMetricCollector:

    def get_sample_metrics(self):
        with app.connection_or_acquire() as conn:
            client = conn.channel().client
            result = client.get('some_metric_key')
            return {'some_metric_key': result}

    def collect(self):
        sample_metrics = self.get_sample_metrics()
        for key, value in sample_metrics.items():
            yield GaugeMetricFamily(key, 'This is a custom metric', value=value)


REGISTRY.register(SomeMetricCollector())

tasks.py

# This is my boilerplate taskbroker app
from my_awesome_app.taskbroker.celery import app
# How it's collecting data from postgres is trivial to this issue.
from my_awesome_app.utility_app.utility import some_value_calculated_from_query


@app.task()
def app_metrics_sync_periodic():
    with app.connection_or_acquire() as conn:
        client = conn.channel().client
        client.set('some_metric_key', some_value_calculated_from_query(), ex=21600)
        return True

I don't think the background data collection in tasks.py is causing the Redis connections to grow exponentially but it's the Django view '/metrics' in views.py which is causing.

Can you please tell me what I am doing wrong here? If there is a better way to read from Redis from a Django view. The Prometheus instance scrapes the Django application every 5s .

Answer 1

This answer is according to my use case and research.

The issue here, according to me, is the fact that each request to /metrics initiates a new thread where the views.py creates new connections in the Celery broker's connection pool.

This can be easily handled by letting Django manage its own Redis connection pool through cache backend and Celery manage its own Redis connection pool and not use each other's connection pools from their respective threads.

Django Side

config.py

# CACHES
# ------------------------------------------------------------------------------
# For more details on options for your cache backend please refer
# https://docs.djangoproject.com/en/3.1/ref/settings/#backend
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://localhost:6379/0",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
        },
    }
}

views.py

from prometheus_client.core import GaugeMetricFamily, REGISTRY
# *: Replacing celery app with Django cache backend
from django.core.cache import cache


class SomeMetricCollector:

    def get_sample_metrics(self):
        # *: This is how you will get the new client, which is still context managed.
        with cache.client.get_client() as client:
            result = client.get('some_metric_key')
            return {'some_metric_key': result}

    def collect(self):
        sample_metrics = self.get_sample_metrics()
        for key, value in sample_metrics.items():
            yield GaugeMetricFamily(key, 'This is a custom metric', value=value)


REGISTRY.register(SomeMetricCollector())

This will ensure that Django will maintain it's Redis connection pool and not cause new connections to be spun up unnecessarily.

Celery Side

tasks.py

# This is my boilerplate taskbroker app
from my_awesome_app.taskbroker.celery import app
# How it's collecting data from postgres is trivial to this issue.
from my_awesome_app.utility_app.utility import some_value_calculated_from_query


@app.task()
def app_metrics_sync_periodic():
    with app.connection_or_acquire() as conn:
        # *: This will force celery to always look into the existing connection pool for connection.
        client = conn.default_channel.client
        client.set('some_metric_key', some_value_calculated_from_query(), ex=21600)
        return True

How do I monitor connections?

• There is a nice prometheus celery exporter which will help you monitor your celery task activity not sure how you can add connection pool and connection monitoring to it.
• The easiest way to manually verify if the connections are growing every time /metrics is hit on the web app, is by:

   $ redis-cli 127.0.0.1:6379> CLIENT LIST...

• The client list command will help you see if the number of connections are growing or not.
• I don't use queues sadly but I would recommend using queues. This is how my worker runs:

   $ celery -A my_awesome_app.taskbroker worker --concurrency=20 -l ERROR -E