[英]How can I notify RxPY observers on separate threads using asyncio?
(注意:此問題的背景非常詳細,但底部有一個SSCCE可以跳過)
我正在嘗試開發基於Python的CLI來與Web服務進行交互。 在我的代碼庫中,我有一個CommunicationService
類,它處理與Web服務的所有直接通信。 它公開了一個received_response
返回屬性的Observable
(從RxPY),其他對象可以訂閱才能收到通知時,接收響應從Web服務了。
我將CLI邏輯基於click
庫,其中一個子命令實現如下:
async def enabled(self, request: str, response_handler: Callable[[str], Tuple[bool, str]]) -> None:
self._generate_request(request)
if response_handler is None:
return None
while True:
response = await self.on_response
success, value = response_handler(response)
print(success, value)
if success:
return value
這里發生的事情(在response_handler
不是None
的情況下)是子命令表現為一個協程,它等待來自Web服務的響應( self.on_response == CommunicationService.received_response
)並從第一個響應中返回一些處理過的值。處理。
我正在嘗試通過創建完全模擬CommunicationService
測試用例來測試CLI的行為。 創建一個假Subject
(可以充當Observable
),並且Observable
CommunicationService.received_response
以返回它。 作為測試的一部分,調用主題的on_next
方法將模擬Web服務響應傳遞回生產代碼:
@when('the communications service receives a response from TestCube Web Service')
def step_impl(context):
context.mock_received_response_subject.on_next(context.text)
我使用單擊'結果回調'函數,該函數在CLI調用結束時被調用並阻塞,直到coroutine(子命令)完成:
@cli.resultcallback()
def _handle_command_task(task: Coroutine, **_) -> None:
if task:
loop = asyncio.get_event_loop()
result = loop.run_until_complete(task)
loop.close()
print('RESULT:', result)
在測試開始時,我運行CliRunner.invoke
來解雇整個shebang。 問題是這是一個阻塞調用,並且會阻塞線程,直到CLI完成並返回結果,如果我需要繼續執行測試線程,這樣做無效,因此可以與它同時生成模擬Web服務響應。
我想我需要做的是使用ThreadPoolExecutor
在新線程上運行CliRunner.invoke
。 這允許測試邏輯在原始線程上繼續並執行上面發布的@when
步驟。 但是, 使用 mock_received_response_subject.on_next
發布的通知 似乎不會觸發執行以在子命令中繼續 。
我相信解決方案將涉及使用RxPY的AsyncIOScheduler
,但我發現這方面的文檔有點稀疏且無益。
下面的代碼片段捕捉了我希望問題的本質。 如果它可以被修改為工作,我應該能夠將相同的解決方案應用於我的實際代碼,以使其按照我的意願運行。
import asyncio
import logging
import sys
import time
import click
from click.testing import CliRunner
from rx.subjects import Subject
web_response_subject = Subject()
web_response_observable = web_response_subject.as_observable()
thread_loop = asyncio.new_event_loop()
@click.group()
def cli():
asyncio.set_event_loop(thread_loop)
@cli.resultcallback()
def result_handler(task, **_):
loop = asyncio.get_event_loop()
result = loop.run_until_complete(task) # Should block until subject publishes value
loop.close()
print(result)
@cli.command()
async def get_web_response():
return await web_response_observable
def test():
runner = CliRunner()
future = thread_loop.run_in_executor(None, runner.invoke, cli, ['get_web_response'])
time.sleep(1)
web_response_subject.on_next('foo') # Simulate reception of web response.
time.sleep(1)
result = future.result()
print(result.output)
logging.basicConfig(
level=logging.DEBUG,
format='%(threadName)10s %(name)18s: %(message)s',
stream=sys.stderr,
)
test()
當前行為
程序在運行時掛起,阻塞在result = loop.run_until_complete(task)
。
驗收標准
程序終止並在stdout
上打印foo
。
基於Vincent的幫助,我對代碼進行了一些更改。
Relay.enabled
(等待來自Web服務的響應以便處理它們的子命令)現在實現如下:
async def enabled(self, request: str, response_handler: Callable[[str], Tuple[bool, str]]) -> None:
self._generate_request(request)
if response_handler is None:
return None
return await self.on_response \
.select(response_handler) \
.where(lambda result, i: result[0]) \
.select(lambda result, index: result[1]) \
.first()
我不太確定await
如何使用RxPY
observables - 它們會在生成的每個元素上將執行返回給調用者,還是僅在observable完成(或錯誤?)時才執行。 我現在知道它是后者,老實說感覺就像是更自然的選擇,並且讓我覺得這個功能的實現感覺更加優雅和反應。
我還修改了生成模擬Web服務響應的測試步驟:
@when('the communications service receives a response from TestCube Web Service')
def step_impl(context):
loop = asyncio.get_event_loop()
loop.call_soon_threadsafe(context.mock_received_response_subject.on_next, context.text)
不幸的是, 這不會起作用 ,因為CLI是在自己的線程中調用的......
@when('the CLI is run with "{arguments}"')
def step_impl(context, arguments):
loop = asyncio.get_event_loop()
if 'async.cli' in context.tags:
context.async_result = loop.run_in_executor(None, context.cli_runner.invoke, testcube.cli, arguments.split())
else:
...
並且CLI在調用時創建自己的線程私有事件循環...
def cli(context, hostname, port):
_initialize_logging(context.meta['click_log.core.logger']['level'])
# Create a new event loop for processing commands asynchronously on.
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
...
我認為我需要的是一種允許我的測試步驟在新線程上調用CLI 然后獲取它使用的事件循環的方法 :
@when('the communications service receives a response from TestCube Web Service')
def step_impl(context):
loop = _get_cli_event_loop() # Needs to be implemented.
loop.call_soon_threadsafe(context.mock_received_response_subject.on_next, context.text)
似乎沒有一種簡單的方法來獲取特定線程為自己創建和使用的事件循環,因此我采用了Victor的建議並asyncio.new_event_loop
以返回我的測試代碼創建和存儲的事件循環:
def _apply_mock_event_loop_patch(context):
# Close any already-existing exit stacks.
if hasattr(context, 'mock_event_loop_exit_stack'):
context.mock_event_loop_exit_stack.close()
context.test_loop = asyncio.new_event_loop()
print(context.test_loop)
context.mock_event_loop_exit_stack = ExitStack()
context.mock_event_loop_exit_stack.enter_context(
patch.object(asyncio, 'new_event_loop', spec=True, return_value=context.test_loop))
我更改了我的“模擬網絡響應收到”測試步驟,以執行以下操作:
@when('the communications service receives a response from TestCube Web Service')
def step_impl(context):
loop = context.test_loop
loop.call_soon_threadsafe(context.mock_received_response_subject.on_next, context.text)
好消息是,當我執行此步驟時,我實際上正在啟動Relay.enabled
協程!
現在唯一的問題是最后的測試步驟,我等待從我自己的線程中執行CLI得到的未來,並驗證CLI是否在stdout
上發送:
@then('the CLI should print "{output}"')
def step_impl(context, output):
if 'async.cli' in context.tags:
loop = asyncio.get_event_loop() # main loop, not test loop
result = loop.run_until_complete(context.async_result)
else:
result = context.result
assert_that(result.output, equal_to(output))
我試着玩弄這一點,但我似乎無法得到context.async_result
(從存儲未來loop.run_in_executor
)很好地過渡done
並返回結果。 在當前的實現中,我得到第一個測試( 1.1
)的錯誤和第二個測試的無限掛起( 1.2
):
@mock.comms @async.cli @wip
Scenario Outline: Querying relay enable state -- @1.1 # testcube/tests/features/relay.feature:45
When the user queries the enable state of relay 0 # testcube/tests/features/steps/relay.py:17 0.003s
Then the CLI should query the web service about the enable state of relay 0 # testcube/tests/features/steps/relay.py:48 0.000s
When the communications service receives a response from TestCube Web Service # testcube/tests/features/steps/core.py:58 0.000s
"""
{'module':'relays','path':'relays[0].enabled','data':[True]}'
"""
Then the CLI should print "True" # testcube/tests/features/steps/core.py:94 0.003s
Traceback (most recent call last):
File "/Users/davidfallah/testcube_env/lib/python3.5/site-packages/behave/model.py", line 1456, in run
match.run(runner.context)
File "/Users/davidfallah/testcube_env/lib/python3.5/site-packages/behave/model.py", line 1903, in run
self.func(context, *args, **kwargs)
File "testcube/tests/features/steps/core.py", line 99, in step_impl
result = loop.run_until_complete(context.async_result)
File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/davidfallah/testcube_env/lib/python3.5/site-packages/click/testing.py", line 299, in invoke
output = out.getvalue()
ValueError: I/O operation on closed file.
Captured stdout:
RECEIVED WEB RESPONSE: {'module':'relays','path':'relays[0].enabled','data':[True]}'
<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py:431]>
@mock.comms @async.cli @wip
Scenario Outline: Querying relay enable state -- @1.2 # testcube/tests/features/relay.feature:46
When the user queries the enable state of relay 1 # testcube/tests/features/steps/relay.py:17 0.005s
Then the CLI should query the web service about the enable state of relay 1 # testcube/tests/features/steps/relay.py:48 0.001s
When the communications service receives a response from TestCube Web Service # testcube/tests/features/steps/core.py:58 0.000s
"""
{'module':'relays','path':'relays[1].enabled','data':[False]}'
"""
RECEIVED WEB RESPONSE: {'module':'relays','path':'relays[1].enabled','data':[False]}'
Then the CLI should print "False" # testcube/tests/features/steps/core.py:94
擰緊所有這些異步多線程的東西,我太愚蠢了。
首先,而不是像這樣描述這樣的場景......
When the user queries the enable state of relay <relay_id>
Then the CLI should query the web service about the enable state of relay <relay_id>
When the communications service receives a response from TestCube Web Service:
"""
{"module":"relays","path":"relays[<relay_id>].enabled","data":[<relay_enabled>]}
"""
Then the CLI should print "<relay_enabled>"
我們這樣描述:
Given the communications service will respond to requests:
"""
{"module":"relays","path":"relays[<relay_id>].enabled","data":[<relay_enabled>]}
"""
When the user queries the enable state of relay <relay_id>
Then the CLI should query the web service about the enable state of relay <relay_id>
And the CLI should print "<relay_enabled>"
實施新的給定步驟:
@given('the communications service will respond to requests')
def step_impl(context):
response = context.text
def publish_mock_response(_):
loop = context.test_loop
loop.call_soon_threadsafe(context.mock_received_response_subject.on_next, response)
# Configure the mock comms service to publish a mock response when a request is made.
instance = context.mock_comms.return_value
instance.send_request.on_next.side_effect = publish_mock_response
繁榮
2 features passed, 0 failed, 0 skipped
22 scenarios passed, 0 failed, 0 skipped
58 steps passed, 0 failed, 0 skipped, 0 undefined
Took 0m0.111s
我可以看到你的代碼有兩個問題:
RxPy
不使用Observable.to_future中的任何一個,因此您必須在運行asyncio事件循環的同一線程中訪問RxPy
對象。 on_completed
時, RxPy
設置未來的結果,以便等待observable返回最后發出的對象。 這意味着您必須同時調用on_next
和on_completed
以await
返回。 這是一個工作示例:
import click
import asyncio
from rx.subjects import Subject
from click.testing import CliRunner
web_response_subject = Subject()
web_response_observable = web_response_subject.as_observable()
main_loop = asyncio.get_event_loop()
@click.group()
def cli():
pass
@cli.resultcallback()
def result_handler(task, **_):
future = asyncio.run_coroutine_threadsafe(task, main_loop)
print(future.result())
@cli.command()
async def get_web_response():
return await web_response_observable
def test():
runner = CliRunner()
future = main_loop.run_in_executor(
None, runner.invoke, cli, ['get_web_response'])
main_loop.call_later(1, web_response_subject.on_next, 'foo')
main_loop.call_later(2, web_response_subject.on_completed)
result = main_loop.run_until_complete(future)
print(result.output, end='')
if __name__ == '__main__':
test()
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.