[英]Use Flask's “app” singleton to Dask Scheduler/Workers
The Case:案子:
We have some time-consuming functional/integration tests that utilize Flask's current_app
for configuration (global variables etc.) and some logging.我们有一些耗时的功能/集成测试,它们利用 Flask 的current_app
进行配置(全局变量等)和一些日志记录。
We are trying to distribute and parallelize those tests on a cluster (for the moment a local "cluster" created from Dask's Docker image .).我们正在尝试在集群上分发和并行化这些测试(目前是从Dask 的 Docker 映像创建的本地“集群”。)。
The Issue(s?):问题?):
Let's assume the following example:让我们假设以下示例:
A time-consuming function:一个耗时的function:
def will_take_my_time(n)
# Add the 'TAKE_YOUR_TIME' in the config in how many seconds you want
time.sleep(current_app.config['TAKE_YOUR_TIME'])
return n
A time-consuming test:耗时的测试:
def need_my_time_test(counter=None):
print(f"Test No. {will_take_my_time(counter)}")
A Flask CLI command that creates a Dask Client
to connect to the cluster and execute 10 tests of need_my_time_test
:一个 Flask CLI 命令,它创建一个 Dask Client
以连接到集群并执行 10 次need_my_time_test
测试:
@app.cli.command()
def itests(extended):
with Client(processes=False) as dask_client:
futures = dask_client.map(need_my_time_test, range(10))
print(f"Futures: {futures}")
print(f"Gathered: {dask_client.gather(futures)}")
EDIT: For convenience let's add an application factory for an easier reproducible example:编辑:为方便起见,让我们添加一个应用程序工厂以获得更容易重现的示例:
def create_app():
app = Flask(__name__)
app.config.from_mapping(
SECRET_KEY='dev',
DEBUG=True,
)
@app.route('/hello')
def hello():
return 'Hello, World!'
@app.cli.command()
def itests(extended):
with Client(processes=False) as dask_client:
futures = dask_client.map(need_my_time_test, range(10))
print(f"Futures: {futures}")
print(f"Gathered: {dask_client.gather(futures)}")
Using the above with flask itests
, we are running into the following error (described here ):将上述内容与flask itests
使用,我们遇到以下错误(在此处描述):
RuntimeError: Working outside of application context. RuntimeError:在应用程序上下文之外工作。
This typically means that you attempted to use functionality that needed to interface with the current application object in some way.这通常意味着您试图以某种方式使用与当前应用程序 object 交互所需的功能。 To solve this, set up an application context with app.app_context().要解决此问题,请使用 app.app_context() 设置应用程序上下文。
We have tried:我们尝试过:
app_context
( app.app_context().push()
) on the app singleton creation.在应用程序 singleton 创建上推送app_context
( app.app_context().push()
)。with current_app.app_context():
on the CLI command and some of the functions that use the current_app
. with current_app.app_context():
在 CLI 命令和一些使用current_app
的函数上。app_context
through a Dask Variable
but it cannot serialize the context.通过 Dask Variable
发送app_context
但它不能序列化上下文。With no avail.无济于事。
The questions:问题:
When using current_app
proxy, it is assumed that the Flask app is created in the same process that the proxy is used.使用current_app
代理时,假设 Flask 应用程序是在使用代理的同一进程中创建的。
This is not the situation when tasks submitted to the workers are run.这不是运行提交给工作人员的任务时的情况。 The tasks are executed isolated away from the Flask app created in the process that submitted the tasks.任务的执行与在提交任务的进程中创建的 Flask 应用程序隔离开来。
In the task, define the flask app and provide the application context there.在任务中,定义 flask 应用程序并在那里提供应用程序上下文。
import time
from flask import Flask
from dask.distributed import Client
def _create_app():
app = Flask(__name__)
app.config.from_mapping(
SECRET_KEY='dev',
DEBUG=True,
TAKE_YOUR_TIME=0.2
)
return app
def will_take_my_time(n):
# Add the 'TAKE_YOUR_TIME' in the config in how many seconds you want
app = _create_app()
with app.app_context():
time.sleep(app.config['TAKE_YOUR_TIME'])
return n
def need_my_time_test(counter=None):
print(f"Test No. {will_take_my_time(counter)}")
def create_app():
app = _create_app()
@app.route('/hello')
def hello():
return 'Hello, World!'
@app.cli.command()
def itests():
with Client(processes=False) as dask_client:
futures = dask_client.map(need_my_time_test, range(10))
print(f"Futures: {futures}")
print(f"Gathered: {dask_client.gather(futures)}")
return app
app = create_app()
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.