[英]Is there a way to use asyncio.Queue in multiple threads?
Let's assume I have the following code:假设我有以下代码:
import asyncio
import threading
queue = asyncio.Queue()
def threaded():
import time
while True:
time.sleep(2)
queue.put_nowait(time.time())
print(queue.qsize())
@asyncio.coroutine
def async():
while True:
time = yield from queue.get()
print(time)
loop = asyncio.get_event_loop()
asyncio.Task(async())
threading.Thread(target=threaded).start()
loop.run_forever()
The problem with this code is that the loop inside async
coroutine is never finishing the first iteration, while queue
size is increasing.这段代码的问题在于
async
协程中的循环永远不会完成第一次迭代,而queue
大小正在增加。
Why is this happening this way and what can I do to fix it?为什么会以这种方式发生这种情况,我可以做些什么来解决它?
I can't get rid of separate thread, because in my real code I use a separate thread to communicate with a serial device, and I haven't find a way to do that using asyncio
.我无法摆脱单独的线程,因为在我的实际代码中,我使用单独的线程与串行设备进行通信,而我还没有找到使用
asyncio
做到这一点的方法。
asyncio.Queue
is not thread-safe , so you can't use it directly from more than one thread. asyncio.Queue
不是线程安全的,因此您不能直接从多个线程使用它。 Instead, you can use janus
, which is a third-party library that provides a thread-aware asyncio
queue:相反,您可以使用
janus
,它是一个提供线程感知asyncio
队列的第三方库:
import asyncio
import threading
import janus
def threaded(squeue):
import time
while True:
time.sleep(2)
squeue.put_nowait(time.time())
print(squeue.qsize())
@asyncio.coroutine
def async(aqueue):
while True:
time = yield from aqueue.get()
print(time)
loop = asyncio.get_event_loop()
queue = janus.Queue(loop=loop)
asyncio.Task(asyncio.ensure_future(queue.async_q))
threading.Thread(target=threaded, args=(queue.sync_q,)).start()
loop.run_forever()
There is also aioprocessing
(full-disclosure: I wrote it), which provides process-safe (and as a side-effect, thread-safe) queues as well, but that's overkill if you're not trying to use multiprocessing
.还有
aioprocessing
(完全公开:我写的),它也提供进程安全(作为副作用,线程安全)队列,但如果您不尝试使用multiprocessing
,那就有点过头了。
Edit编辑
As pointed it out in other answers, for simple use-cases you can use loop.call_soon_threadsafe
to add to the queue, as well.正如其他答案中指出的那样,对于简单的用例,您也可以使用
loop.call_soon_threadsafe
添加到队列中。
BaseEventLoop.call_soon_threadsafe
is at hand. BaseEventLoop.call_soon_threadsafe
就在眼前。 See asyncio
doc for detail.有关详细信息,请参阅
asyncio
文档。
Simply change your threaded()
like this:只需像这样更改
threaded()
:
def threaded():
import time
while True:
time.sleep(1)
loop.call_soon_threadsafe(queue.put_nowait, time.time())
loop.call_soon_threadsafe(lambda: print(queue.qsize()))
Here's a sample output:这是一个示例输出:
0
1443857763.3355968
0
1443857764.3368602
0
1443857765.338082
0
1443857766.3392274
0
1443857767.3403943
If you do not want to use another library you can schedule a coroutine from the thread.如果您不想使用另一个库,您可以从线程中安排一个协程。 Replacing the
queue.put_nowait
with the following works fine.用以下替换
queue.put_nowait
工作正常。
asyncio.run_coroutine_threadsafe(queue.put(time.time()), loop)
The variable loop
represents the event loop in the main thread.变量
loop
代表主线程中的事件循环。
EDIT:编辑:
The reason why your async
coroutine is not doing anything is that the event loop never gives it a chance to do so.你的
async
协程没有做任何事情的原因是事件循环从来没有给它机会这样做。 The queue object is not threadsafe and if you dig through the cpython code you find that this means that put_nowait
wakes up consumers of the queue through the use of a future with the call_soon
method of the event loop.队列对象不是线程安全的,如果您深入研究 cpython 代码,您会发现这意味着
put_nowait
通过使用带有事件循环的call_soon
方法的未来来唤醒队列的使用者。 If we could make it use call_soon_threadsafe
it should work.如果我们可以让它使用
call_soon_threadsafe
它应该可以工作。 The major difference between call_soon
and call_soon_threadsafe
, however, is that call_soon_threadsafe
wakes up the event loop by calling loop._write_to_self()
.然而,
call_soon
和call_soon_threadsafe
之间的主要区别在于call_soon
通过调用loop._write_to_self()
call_soon_threadsafe
唤醒事件循环。 So let's call it ourselves:所以让我们自己称它为:
import asyncio
import threading
queue = asyncio.Queue()
def threaded():
import time
while True:
time.sleep(2)
queue.put_nowait(time.time())
queue._loop._write_to_self()
print(queue.qsize())
@asyncio.coroutine
def async():
while True:
time = yield from queue.get()
print(time)
loop = asyncio.get_event_loop()
asyncio.Task(async())
threading.Thread(target=threaded).start()
loop.run_forever()
Then, everything works as expected.然后,一切都按预期进行。
As for the threadsafe aspect of accessing shared objects, asyncio.queue
uses under the hood collections.deque
which has threadsafe append
and popleft
.至于访问共享对象的线程安全方面,
asyncio.queue
在引擎盖下使用collections.deque
,它具有线程安全append
和popleft
。 Maybe checking for queue not empty and popleft is not atomic, but if you consume the queue only in one thread (the one of the event loop) it could be fine.也许检查队列不为空并且 popleft 不是原子的,但是如果您仅在一个线程(事件循环之一)中使用队列,那可能没问题。
The other proposed solutions, loop.call_soon_threadsafe
from Huazuo Gao's answer and my asyncio.run_coroutine_threadsafe
are just doing this, waking up the event loop.其他提出的解决方案,
loop.call_soon_threadsafe
从化做高的回答和我的asyncio.run_coroutine_threadsafe
只是在做这个,醒来事件循环。
What about just using threading.Lock with asyncio.Queue?仅将 threading.Lock 与 asyncio.Queue 一起使用怎么样?
class ThreadSafeAsyncFuture(asyncio.Future):
""" asyncio.Future is not thread-safe
https://stackoverflow.com/questions/33000200/asyncio-wait-for-event-from-other-thread
"""
def set_result(self, result):
func = super().set_result
call = lambda: func(result)
self._loop.call_soon_threadsafe(call) # Warning: self._loop is undocumented
class ThreadSafeAsyncQueue(queue.Queue):
""" asyncio.Queue is not thread-safe, threading.Queue is not awaitable
works only with one putter to unlimited-size queue and with several getters
TODO: add maxsize limits
TODO: make put corouitine
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.lock = threading.Lock()
self.loop = asyncio.get_event_loop()
self.waiters = []
def put(self, item):
with self.lock:
if self.waiters:
self.waiters.pop(0).set_result(item)
else:
super().put(item)
async def get(self):
with self.lock:
if not self.empty():
return super().get()
else:
fut = ThreadSafeAsyncFuture()
self.waiters.append(fut)
result = await fut
return result
See also - asyncio: Wait for event from other thread另请参阅 - asyncio:等待来自其他线程的事件
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.