[英]slackclient OSError: [Errno 24] Too many open files
我在我的應用程序上添加了slackclient以獲得常規通知。 該應用程序在AWS Linux和docker容器(FROM python:3.6.6-stretch)中執行。
每當我嘗試在大約1000個循環中執行它,但該過程在1000個循環后失敗,並且出現以下錯誤。
我已經檢查了關於stackoverflow的其他情況,但是對我來說不起作用。 希望我能從您的回答中得到一些啟示。 我個人想知道為什么會這樣並解決核心問題,而不僅僅是繞道而行。
錯誤
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 321, in ssl_wrap_socket
OSError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 323, in ssl_wrap_socket
urllib3.exceptions.SSLError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.update (Caused by SSLError(OSError(24, 'Too many open files'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/scraping/notify.py", line 50, in update
File "/home/scraping/notify.py", line 70, in _api_call
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/client.py", line 184, in api_call
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/server.py", line 349, in api_call
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 84, in do
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 116, in post_http_request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 116, in post
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 60, in request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
requests.exceptions.SSLError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.update (Caused by SSLError(OSError(24, 'Too many open files'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
OSError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.postMessage (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 33, in <module>
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 127, in Fire
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 366, in _Fire
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 542, in _CallCallable
File "main.py", line 29, in fasten
File "/home/scraping/tasks/apps_info.py", line 199, in fasten
File "/home/scraping/pipe.py", line 144, in _notify
File "/home/scraping/notify.py", line 53, in update
File "/home/scraping/notify.py", line 46, in resend
File "/home/scraping/notify.py", line 70, in _api_call
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/client.py", line 184, in api_call
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/server.py", line 349, in api_call
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 84, in do
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 116, in post_http_request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 116, in post
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 60, in request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.postMessage (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files',))
SlackClient部分
import errno
import time
import urllib3
from datetime import datetime
from typing import List, Text
import requests
from slackclient import SlackClient
import config
from .utils import get_utc_date
class SlackHelper:
def __init__(self):
self._date_format = "%Y-%m-%d %H-%M-%S"
self._sc = SlackClient(config.SLACK_TOKEN)
self._channel = config.SLACK_CHANNEL
self._user_name = config.SLACK_USERNAME
self._icon_emoji = config.SLACK_MSG_EMOJI
self._ts = None
self._ch_encoded = None
self._created_at = get_utc_date(_format=self._date_format)
def send(self, *args, **kw):
return self._api_call(*args, **kw)
def resend(self, resend_cnt: int = 0, *args, **kw):
try:
self._ts = None
kw['method'] = 'chat.postMessage'
except (urllib3.exceptions.NewConnectionError,
urllib3.exceptions.MaxRetryError,
requests.exceptions.ConnectionError) as err:
if resend_cnt < 5:
time.sleep(60)
resend_cnt += 1
self.resend(resend_cnt, *args, **kw)
raise err
return self._api_call(*args, **kw)
def update(self, *args, **kw):
try:
return self._api_call(*args, **kw)
except (requests.exceptions.ConnectionError,
requests.exceptions.SSLError):
self.resend(*args, **kw)
except OSError as err:
if err.errno == errno.EMFILE:
return self.resend(*args, **kw)
raise err
def _api_call(self, title_prefix: Text, msg: Text,
alarm: Text = 'good', method: Text = 'chat.postMessage'):
attachments = self._get_attachments(title_prefix, msg, alarm)
response = self._sc.api_call(
method=method,
channel=(
self._ch_encoded if method == 'chat.update'
else self._channel),
attachments=[attachments],
username=self._user_name,
icon_emoji=self._icon_emoji,
ts=self._ts if method == 'chat.update' else None)
if self._check_retry(response):
delay = int(response["headers"]["Retry-After"])
time.sleep(delay)
return self._api_call(title_prefix, msg, alarm, method)
if response["ok"]:
if not self._ts:
self._ts = response.get('ts')
self._ch_encoded = response.get('channel')
return response.get('ok', True)
@staticmethod
def _check_retry(response):
if response["ok"] is False:
if response["headers"].get("Retry-After", False):
delay = int(response["headers"]["Retry-After"])
if delay < 60:
return True
return False
def _get_attachments(self, title_prefix, msg, alarm):
now = get_utc_date(_format=self._date_format)
return {
"title": f"Created : {self._created_at}\n{title_prefix} : {now}",
"text": msg,
"mrkdwn_in": ["text"],
"color": f"{alarm}"
}
ENV
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15698
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15698
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
嘗試使用--ulimit
參數運行 --ulimit
容器
docker run -d --ulimit nofile=98304:98304 ...
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.