[英]Tornado “error: [Errno 24] Too many open files” error
我和Tornado一起工作了很多,但這是我第一次遇到這種錯誤。 我一直在研究一個非常基本的URL縮短器。 URL由不同的應用程序放入數據庫,這只是從MongoDB商店讀取URL並重定向客戶端。 在我編寫了基本代碼之后,我設置了一個簡單的'Siege'測試,在運行圍攻大約30秒后(使用siege -c 64 -t 5m -r 1 http://example.com/MKy
對抗4個應用程序線程)我開始得到500個響應。 查看錯誤日志,我看到了這個;
ERROR:root:500 GET /MKy (127.0.0.1) 2.05ms
ERROR:root:Exception in I/O handler for fd 4
Traceback (most recent call last):
File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/ioloop.py", line 309, in start
File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/netutil.py", line 314, in accept_handler
File "/opt/python2.7/lib/python2.7/socket.py", line 200, in accept
error: [Errno 24] Too many open files
ERROR:root:Uncaught exception GET /MKy (127.0.0.1)
HTTPRequest(protocol='http', host='shortener', method='GET', uri='/MKy', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': 'shortener', 'Accept-Encoding': 'gzip', 'X-Real-Ip': '94.23.155.32', 'X-Forwarded-For': '94.23.155.32', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'JoeDog/1.00 [en] (X11; I; Siege 2.66)'})
Traceback (most recent call last):
File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/web.py", line 1040, in wrapper
File "main.py", line 58, in get
File "main.py", line 21, in dbmongo
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 349, in __init__
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 510, in __find_master
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 516, in __try_node
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/database.py", line 301, in command
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/collection.py", line 441, in find_one
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 539, in loop
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 560, in _refresh
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 620, in __send_message
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 735, in _send_message_with_response
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 591, in __stream
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 200, in get_stream
File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 559, in __connect
AutoReconnect: could not connect to [('127.0.0.1', 27017)]
重要的(我猜);
錯誤:[Errno 24]打開的文件太多
代碼; (這很簡單)
import tornado.ioloop
import tornado.web
import tornado.escape
import apymongo
import time
import sys
#Useful stuff (Connect to Mongo)
class setup(tornado.web.RequestHandler):
def dbmongo(self):
if not hasattr(self, '_dbmongo'):
self._dbmongo = apymongo.Connection("127.0.0.1", 27017)
return self._dbmongo
#Basic method to lookup URLs from Mongo and redirect accordingly
class expand(setup):
@tornado.web.asynchronous
def get(self, url):
self.mongo = self.dbmongo()
#Lookup the URL
cursor = self.mongo.rmgshortlinks.links.find_one({'short':url}, self.direct)
def direct(self, response):
if response == None:
self.send_error(404)
self.finish()
return
link = tornado.escape.url_unescape(response['long'])
#Bounce the client
self.write("<!DOCTYPE html><html><head><meta charset=\"UTF-8\" /><meta http-equiv=\"refresh\" content=\"0;URL="+link+"\"</head><body><a href=\""+link+"\">Click Here</a></body></html>")
self.finish();
#Define the URL routes
application = tornado.web.Application([
(r"/([a-zA-Z0-9]+)", expand)
])
#Start the server
if __name__ == "__main__":
listening_port = int(sys.argv[1])
if listening_port > 0:
application.listen(listening_port)
tornado.ioloop.IOLoop.instance().start()
else:
sys.stderr.write("No port specified!")
我正在使用的開發服務器有8個內核和64GB內存,運行RedHat Enterprise Linux 5和Python 2.6。 我之前從未遇到過Tornado / Async Mongo應用程序的這些問題。
可能有用的信息;
[root@puma ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31374
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31374
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
(打開文件只設置為1024,但我認為這已經足夠了)
Tornado / Apymongo沒有正確關閉連接嗎? 應用程序位於NGINX后面但使用HTTP連接,Apymongo應該通過TCP連接,但可能使用套接字。 即便如此,它應該是共享/池連接不應該嗎?
按照建議,將應用程序移動到我們的一個測試服務器上,最大打開文件限制為61440,在圍攻中運行約30秒后出現相同的錯誤。
很簡單,RequestHandler對象是為每個請求實例化的。 這意味着您要保存的緩存對象位於RequestHandler(例如,expand)對象上。
如果要向dbmongo(...)函數添加一個簡單的“print'CREATED!'”,您會看到它是在每個GET請求上創建的。
您需要做的是將處理程序附加到類對象,或根據需要附加“全局”,但最好的情況是將它放在Tornado Application對象上。
簡單:
class setup(tornado.web.RequestHandler):
@classmethod
def dbmongo(cls):
if not hasattr(cls, '_dbmongo'):
cls._dbmongo = apymongo.Connection("127.0.0.1", 27017)
return cls._dbmongo
第二種方法就是在文件中使其成為全局:
dbmongo_connection = None
def dbmongo():
if not dbmongo_connection:
dbmongo_connection = apymongo.Connection("127.0.0.1", 27017)
return dbmongo_connection
這兩個都有同樣的問題,如果你有很多想要使用數據庫連接的類,那么分享它就更難了。 由於DB是一個共享實體,您可能需要一個用於整個應用程序的實體。
class MongoMixin(object):
def mongodb(self):
if not hasattr(self.application, 'mongodb'):
self.application.mongodb = apymongo.Connection(self.application.settings.get("mongohost", "127.0.0.1"), 27017)
return self.application.mongodb
class expand(tornado.web.RequestHandler, MongoMixin):
def get(self):
db = self.mongodb()
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.