[英]Tornado concurrency errors running multiple processes together with a process pool executor
我正在嘗試運行多個進程,同時使用concurrent.futures.ProcessPoolExecutor
來運行CPU密集型作業。 前幾個請求很愉快,但隨后從concurrent.futures.process
引發了KeyError
,服務器掛起。
這是龍卷風中的一個錯誤嗎?
這是我剝離代碼的最簡單形式。
服務器:
# -*- coding: utf-8 -*-
"""
server runs 2 processes and does job on a ProcessPoolExecutor
"""
import tornado.web
import tornado.ioloop
import tornado.gen
import tornado.options
import tornado.httpserver
from concurrent.futures import ProcessPoolExecutor
class MainHandler(tornado.web.RequestHandler):
executor = ProcessPoolExecutor(1)
@tornado.gen.coroutine
def post(self):
num = int(self.request.body)
result = yield self.executor.submit(pow, num, 2)
self.finish(str(result))
application = tornado.web.Application([
(r"/", MainHandler),
])
def main():
tornado.options.parse_command_line()
server = tornado.httpserver.HTTPServer(application)
server.bind(8888)
server.start(2)
tornado.ioloop.IOLoop.instance().start()
if __name__ == '__main__':
main()
客戶:
# -*- coding: utf-8 -*-
"""
client
"""
from tornado.httpclient import AsyncHTTPClient
from tornado.gen import coroutine
from tornado.ioloop import IOLoop
@coroutine
def remote_compute(num):
rsp = yield AsyncHTTPClient().fetch(
'http://127.0.0.1:8888', method='POST', body=str(num))
print 'result:', rsp.body
IOLoop.instance().run_sync(lambda: remote_compute(10))
錯誤回溯
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.7_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/local/Cellar/python/2.7.7_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/cliffxuan/.virtualenvs/executor/lib/python2.7/site-packages/concurrent/futures/process.py", line 216, in _queue_management_worker
work_item = pending_work_items[result_item.work_id]
KeyError: 0
這與使用server.start(2)
啟動具有多個進程的tornado
服務器時tornado
和concurrent.futures
之間的交互有關。 這在內部使用os.fork()
來創建兩個進程。 因為您將Executor
聲明為類變量,所以在執行MainHandler
類本身之前,在server.start()
實際運行之前,它會被實例化。 這意味着兩個進程最終共享一個(盡管是分叉的) ProcessPoolExecutor
實例。 這導致了一些奇怪的事情 - 每個進程都在Executor
中獲得了大多數數據結構的寫時復制版本,但它們最終實際上共享了相同的工作進程。
ProcessPoolExecutor
不支持在這樣的進程之間共享,因此在第二個進程嘗試使用Executor
。 您可以通過僅在fork
發生后創建Executor
來解決它:
class MainHandler(tornado.web.RequestHandler):
executor = None # None for now
@tornado.gen.coroutine
def post(self):
num = int(self.request.body)
result = yield self.executor.submit(pow, num, 2)
self.finish(str(result))
application = tornado.web.Application([
(r"/", MainHandler),
])
def main():
tornado.options.parse_command_line()
server = tornado.httpserver.HTTPServer(application)
server.bind(8889)
server.start(2) # We fork here
MainHandler.executor = ProcessPoolExecutor(1) # Now we can create the Executor
tornado.ioloop.IOLoop.instance().start()
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.