Django too many open files

Question

I'm getting "Too many open files" in my django application. I think it's related to the logging system, because with every request a new descriptor is created for file stage.log.

The handler python_logging_rabbitmq.RabbitMQHandler belongs to this package python-logging-rabbitmq

Any help? Thanks.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'request_id': {
            '()': 'log_request_id.filters.RequestIDFilter'
        },
        'ip': {
            '()': 'audit.ip_address.filters.IPAddressFilter'
        }
    },
    'formatters': {
        'standard': {
            'format': STANDARD_FORMAT
        },
        'compact': {
            'format': COMPACT_FORMAT
        },
        'json': {
            '()': 'pythonjsonlogger.jsonlogger.JsonFormatter',
            'fmt': JSON_FORMAT
        }
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filters': ['request_id', 'ip'],
            'filename': BASE_DIR + '/logs/stage.log',
            'formatter': 'standard',
            'backupCount': LOG_FILE_BACKUP_COUNT,
            'maxBytes': LOG_FILE_MAX_SIZE
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'filters': ['request_id', 'ip'],
            'formatter': 'compact'
        },
        'rabbit': {
            'level': 'DEBUG',
            'class': 'python_logging_rabbitmq.RabbitMQHandler',
            'formatter': 'json',
            'host': RABBIT_HOST,
            'username': RABBIT_USER,
            'password': RABBIT_PWD,
            'exchange': 'log',
            'fields': {
                'origin': 'main-api',
                'env': 'stage'
            },
            'fields_under_root': True
        }
    },
    'loggers': {
        'app': {
            'handlers': ['console', 'file', 'rabbit'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'django.request': {
            'handlers': ['console', 'file'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'requests.packages.urllib3': {
            'handlers': ['console', 'file'],
            'level': 'DEBUG',
            'propagate': True,
        }
    }
}

Answer 1

Use ulimit -n to check your system limit. In my machine the value is 65536.

And then you can set a greater value.

ulimit -n 70000

Or:

sudo echo 70000 > /proc/sys/fs/file-max