堆栈内存溢出
  简体   繁体   English

slackclient OSError：[Errno 24]打开的文件太多

[英]slackclient OSError: [Errno 24] Too many open files

 原文  2018-12-06 16:56:03  0   1    python/ linux/ slack/ urllib3/ eoserror

问题描述

I add slackclient on my app to get regular notification. 我在我的应用程序上添加了slackclient以获得常规通知。 The app is executed on AWS Linux and in a docker container (FROM python:3.6.6-stretch). 该应用程序在AWS Linux和docker容器（FROM python：3.6.6-stretch）中执行。

Whenever I tried to execute it over about 1000 loops but the process failed just after 1000 loops and I got the following errors. 每当我尝试在大约1000个循环中执行它，但该过程在1000个循环后失败，并且出现以下错误。

I already checked other cases on stackoverflow but it doesn't work for me. 我已经检查了关于stackoverflow的其他情况，但是对我来说不起作用。 Hope that I can get some insight from the answer of yours. 希望我能从您的回答中得到一些启示。 I personally want to know why this happens and solve the core problems, not just detour it. 我个人想知道为什么会这样并解决核心问题，而不仅仅是绕道而行。

Error 错误 

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 321, in ssl_wrap_socket
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 323, in ssl_wrap_socket
urllib3.exceptions.SSLError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.update (Caused by SSLError(OSError(24, 'Too many open files'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/scraping/notify.py", line 50, in update
  File "/home/scraping/notify.py", line 70, in _api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/client.py", line 184, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/server.py", line 349, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 84, in do
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 116, in post_http_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 116, in post
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 60, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
requests.exceptions.SSLError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.update (Caused by SSLError(OSError(24, 'Too many open files'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
  File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.postMessage (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 33, in <module>
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 127, in Fire
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 366, in _Fire
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 542, in _CallCallable
  File "main.py", line 29, in fasten
  File "/home/scraping/tasks/apps_info.py", line 199, in fasten
  File "/home/scraping/pipe.py", line 144, in _notify
  File "/home/scraping/notify.py", line 53, in update
  File "/home/scraping/notify.py", line 46, in resend
  File "/home/scraping/notify.py", line 70, in _api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/client.py", line 184, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/server.py", line 349, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 84, in do
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 116, in post_http_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 116, in post
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 60, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.postMessage (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files',))

SlackClient Part SlackClient部分 

import errno
import time
import urllib3
from datetime import datetime
from typing import List, Text

import requests
from slackclient import SlackClient

import config
from .utils import get_utc_date


class SlackHelper:
    def __init__(self):
        self._date_format = "%Y-%m-%d %H-%M-%S"
        self._sc = SlackClient(config.SLACK_TOKEN)
        self._channel = config.SLACK_CHANNEL
        self._user_name = config.SLACK_USERNAME
        self._icon_emoji = config.SLACK_MSG_EMOJI
        self._ts = None
        self._ch_encoded = None
        self._created_at = get_utc_date(_format=self._date_format)

    def send(self, *args, **kw):
        return self._api_call(*args, **kw)

    def resend(self, resend_cnt: int = 0, *args, **kw):
        try:
            self._ts = None
            kw['method'] = 'chat.postMessage'
        except (urllib3.exceptions.NewConnectionError,
                urllib3.exceptions.MaxRetryError,
                requests.exceptions.ConnectionError) as err:
            if resend_cnt < 5:
                time.sleep(60)
                resend_cnt += 1
                self.resend(resend_cnt, *args, **kw)
            raise err
        return self._api_call(*args, **kw)

    def update(self, *args, **kw):
        try:
            return self._api_call(*args, **kw)
        except (requests.exceptions.ConnectionError,
                requests.exceptions.SSLError):
            self.resend(*args, **kw)
        except OSError as err:
            if err.errno == errno.EMFILE:
                return self.resend(*args, **kw)
            raise err

    def _api_call(self, title_prefix: Text, msg: Text,
                alarm: Text = 'good', method: Text = 'chat.postMessage'):
        attachments = self._get_attachments(title_prefix, msg, alarm)
        response = self._sc.api_call(
            method=method,
            channel=(
                self._ch_encoded if method == 'chat.update'
                else self._channel),
            attachments=[attachments],
            username=self._user_name,
            icon_emoji=self._icon_emoji,
            ts=self._ts if method == 'chat.update' else None)

        if self._check_retry(response):
            delay = int(response["headers"]["Retry-After"])
            time.sleep(delay)
            return self._api_call(title_prefix, msg, alarm, method)
        if response["ok"]:
            if not self._ts:
                self._ts = response.get('ts')
            self._ch_encoded = response.get('channel')
        return response.get('ok', True)

    @staticmethod
    def _check_retry(response):
        if response["ok"] is False:
            if response["headers"].get("Retry-After", False):
                delay = int(response["headers"]["Retry-After"])
                if delay < 60:
                    return True
        return False

    def _get_attachments(self, title_prefix, msg, alarm):
        now = get_utc_date(_format=self._date_format)
        return {
            "title": f"Created : {self._created_at}\n{title_prefix} : {now}",
            "text": msg,
            "mrkdwn_in": ["text"],
            "color": f"{alarm}"
        }

ENV ENV 

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15698
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15698
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

1 个解决方案

解决方案1
 0  2018-12-06 19:54:26

尝试使用--ulimit参数运行 --ulimit容器

docker run -d --ulimit nofile=98304:98304 ...

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 OSError：[Errno 24]打开的文件太多 - OSError: [Errno 24] Too many open files OSError: [Errno 24] 打开的文件太多； 在 python； 难以调试 - OSError: [Errno 24] Too many open files; in python; difficult to debug OSError: [Errno 24] 打开的文件太多 - OS Mojave - OSError: [Errno 24] Too many open files - OS Mojave OSError：[Errno 24]太多打开的文件python，ubuntu - OSError: [Errno 24] Too many open files python , ubuntu OSError: [Errno 24] 使用 Nibabel 打开的文件太多 - OSError: [Errno 24] Too many open files using Nibabel OSError: [Errno 24] 通过 Django admin 上传 9000+ csv 个文件时打开的文件太多 - OSError: [Errno 24] Too many open files when uploading 9000+ csv files through Django admin OSError: [Errno 24] 打开的文件太多：&#39;/dev/null&#39; 或没有 IP 显示在我连接的网络上 - OSError: [Errno 24] Too many open files: '/dev/null' or No IP is showing on my connected network OSError: [Errno 24] 在 tensorflow-federated 中训练差分隐私时打开的文件过多 - OSError: [Errno 24] Too many open files when training differential privacy in tensorflow-federated OSError：[Errno 24]在Twisted中使用Reactor.run（）时打开的文件过多 - OSError: [Errno 24] Too many open files when using reactor.run() in Twisted “ OSError：[Errno 24]打开的文件太多：”和来自aiocoap context.request（）的套接字 - “OSError: [Errno 24] Too many open files:” with sockets from aiocoap context.request()
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM