[英]Python asyncio: Running subprocess_exec on a worker thread
So I'm using the Python asyncio
module (on Linux) to launch a child process and then asynchronously monitor it. 因此,我正在使用Python
asyncio
模块(在Linux上)启动子进程,然后对其进行异步监视。 My code works fine... when run on the main thread . 我的代码在主线程上运行时工作正常。 But when I run it on a worker thread, it hangs, and the
process_exited
callback is never invoked. 但是,当我在辅助线程上运行它时,它将挂起,并且永远不会调用
process_exited
回调。
I suspect this may actually be some kind of undocumented defect or issue with running subprocess_exec
on a worker thread, likely having to do with how the implementation handles signals in a background thread. 我怀疑这实际上可能是某种未记录的缺陷,或者是在工作线程上运行
subprocess_exec
问题,可能与实现如何处理后台线程中的信号有关。 But it could also just be me screwing things up. 但这也可能只是我搞砸了。
A simple, reproducible example is as follows: 一个简单的,可复制的示例如下:
class MyProtocol(asyncio.SubprocessProtocol):
def __init__(self, done_future):
super().__init__()
self._done_future = done_future
def pipe_data_received(self, fd, data):
print("Received:", len(data))
def process_exited(self):
print("PROCESS EXITED!")
self._done_future.set_result(None)
def run(loop):
done_future = asyncio.Future(loop = loop)
transport = None
try:
transport, protocol = yield from loop.subprocess_exec(
lambda : MyProtocol(done_future),
"ls",
"-lh",
stdin = None
)
yield from done_future
finally:
if transport: transport.close()
return done_future.result()
def run_loop():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop) # bind event loop to current thread
try:
return loop.run_until_complete(run(loop))
finally:
loop.close()
So here, I setup an asyncio
event loop to execute the shell command ls -lh
, and then trigger a callback for when data is received from the subprocess, and another callback for when the subprocess exits. 因此,在这里,我设置了一个
asyncio
事件循环来执行shell命令ls -lh
,然后为从子流程接收到数据时触发一个回调,为子流程退出时触发另一个回调。
If I just call run_loop()
directly in the main thread of a Python program, everything goes fine. 如果我只是直接在Python程序的主线程中调用
run_loop()
,那么一切都会顺利进行。 But if I say: 但是如果我说:
t = threading.Thread(target = run_loop)
t.start()
t.join()
Then what happens is that the pipe_data_received()
callback is invoked successfully, but process_exited()
is never invoked, and the program just hangs. 然后发生的事情是成功调用了
pipe_data_received()
回调,但是从未调用过process_exited()
,并且程序仅挂起。
After Googling around and looking at the asyncio
source code for the implementation of unix_events.py
, I discovered it might be necessary to manually attach my event loop to the global "child watcher" object, as follows: 在四处
asyncio
并查看了asyncio
的实现的asyncio
源代码unix_events.py
,我发现可能有必要将事件循环手动附加到全局“ child watcher”对象,如下所示:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop) # bind event loop to current thread
asyncio.get_child_watcher().attach_loop(loop)
Apparently, the child watcher is an (undocumented) object that is responsible for calling waitpid
under the hood (or something like that). 显然,儿童观察者是一个(未记录的)对象,负责在内部调用
waitpid
(或类似的东西)。 But when I tried this, and ran run_event_loop()
in a background thread, I got the error: 但是,当我尝试这样做并在后台线程中运行
run_event_loop()
时,出现了错误:
File "/usr/lib/python3.4/asyncio/unix_events.py", line 77, in add_signal_handler
raise RuntimeError(str(exc))
RuntimeError: set_wakeup_fd only works in main thread
So here it looks like the implementation actually does a check to make sure that signal handlers can only be used on the main thread, leading me to believe that in the current implementation, using subprocess_exec
on a background thread is in fact, simply impossible without changing the Python source code itself. 因此,这里看起来该实现实际上进行了检查,以确保只能在主线程上使用信号处理程序,这使我相信,在当前实现中,实际上在后台线程上使用
subprocess_exec
是完全不可能的,而无需更改Python源代码本身。
Am I correct about this? 我对此是否正确? Sadly, the
asyncio
module is very under-documented, so it's hard for me to be confident about my conclusion here. 可悲的是,
asyncio
模块的文档非常少,所以我很难对这里的结论充满信心。 I may simply be doing something wrong. 我可能做错了什么。
Handling subprocesses in a worker thread is fine as long as an asyncio loop is running in the main thread with its child watcher instanciated: 只要在主线程中运行asyncio循环且其子监视程序实例化,就可以在辅助线程中处理子进程:
asyncio.get_child_watcher()
loop = asyncio.get_event_loop()
coro = loop.run_in_executor(None, run_loop)
loop.run_until_complete(coro)
See this post and the documentation . 请参阅这篇文章和文档 。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.