Python Logging getting error: Too many open files

Question

My situation is, I have a configuration file that, obviously, is imported/called every time the system runs.

In my configuration file I have a dict with the logging's configurations and in the beginning of the program I get this configuration and do the logging.config.dictConfig() with the dict.

My system run in subprocess with at least 100 workers using the RQ. Sometimes, when the system is running, I keep getting the error:

Traceback (most recent call last):
File "/home/manutencao/Heimdall/heimdall/worker.py", line 146, in heimdall_tradutor
File "/usr/local/lib/python3.5/logging/__init__.py", line 1279, in info
File "/usr/local/lib/python3.5/logging/__init__.py", line 1415, in _log
File "/usr/local/lib/python3.5/logging/__init__.py", line 1425, in handle
File "/usr/local/lib/python3.5/logging/__init__.py", line 1487, in callHandlers
File "/usr/local/lib/python3.5/logging/__init__.py", line 855, in handle
File "/home/manutencao/Heimdall/heimdall/logger.py", line 28, in emit
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 69, in __init__
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 133, in get_logger
File "/usr/local/lib/python3.5/logging/handlers.py", line 150, in __init__
File "/usr/local/lib/python3.5/logging/handlers.py", line 57, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1008, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1037, in _open
OSError: [Errno 24] Too many open files: '/home/manutencao/Ratatosk/ratatosk/JobRQ.log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/manutencao/.virtualenvs/heimdall/lib/python3.5/site-packages/rq/worker.py", line 588, in perform_job
File "/home/manutencao/.virtualenvs/heimdall/lib/python3.5/site-packages/rq/job.py", line 498, in perform
File "/home/manutencao/Heimdall/heimdall/worker.py", line 195, in heimdall_tradutor
File "/usr/local/lib/python3.5/logging/__init__.py", line 1314, in exception
File "/usr/local/lib/python3.5/logging/__init__.py", line 1308, in error
File "/usr/local/lib/python3.5/logging/__init__.py", line 1415, in _log
File "/usr/local/lib/python3.5/logging/__init__.py", line 1425, in handle
File "/usr/local/lib/python3.5/logging/__init__.py", line 1487, in callHandlers
File "/usr/local/lib/python3.5/logging/__init__.py", line 855, in handle
File "/home/manutencao/Heimdall/heimdall/logger.py", line 28, in emit
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 69, in __init__
File "/home/manutencao/Ratatosk/ratatosk/__init__.py", line 133, in get_logger
File "/usr/local/lib/python3.5/logging/handlers.py", line 150, in __init__
File "/usr/local/lib/python3.5/logging/handlers.py", line 57, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1008, in __init__
File "/usr/local/lib/python3.5/logging/__init__.py", line 1037, in _open
OSError: [Errno 24] Too many open files: '/home/manutencao/Ratatosk/ratatosk/JobRQ.log'

I have a custom logger handler:

class RQHandler(logging.Handler):  # Inherit from logging.Handler
    def __init__(
        self, formatter=JSONFormatter(), level=logging.NOTSET,
        connection_pool=None
    ):
        # run the regular Handler __init__
        logging.Handler.__init__(self, level)
        self.formatter = formatter
        self.connection_pool = connection_pool

    def emit(self, record):
        # record.message is the log message
        ratatosk = Ratatosk(
            all_queues={'huginn_log': [1, 1]},
            debug=configuracoes['level_log'],
            REDIS_HOST=conf_redis['host'],
            REDIS_PORT=conf_redis['port'],
            REDIS_DB=conf_redis['db'],
            connection_pool=(
                self.connection_pool or conf_redis.get('connection_pool')
            )
        )
        ratatosk.enqueue(
            'huginn_log',
            'huginn.service.register_log',
            (self.format(record)),
        )

Basically, I just get the logger and add it to a RQ Queue.

I have read that this error is caused when you have too many handlers in the logging. Is that happening because I added the logging.config.dictConfig() in the beginning of my code?

Thanks in advance

Answer 1

It most likely means there is an error with an unclosed connection to Redis or file in the implementation of RataTosk (a socket counts as an open file in linux). You can increase your file limits in linux if you are confident in the implementation:

ulimit -Hn
ulimit -Sn
# increase the fs.file-max in /etc/sysctl.conf to *2 of greater of the two
# restart the process hosting your python app