The Case:
We have some time-consuming functional/integration tests that utilize Flask's current_app
for configuration (global variables etc.) and some logging.
We are trying to distribute and parallelize those tests on a cluster (for the moment a local "cluster" created from Dask's Docker image .).
The Issue(s?):
Let's assume the following example:
A time-consuming function:
def will_take_my_time(n)
# Add the 'TAKE_YOUR_TIME' in the config in how many seconds you want
time.sleep(current_app.config['TAKE_YOUR_TIME'])
return n
A time-consuming test:
def need_my_time_test(counter=None):
print(f"Test No. {will_take_my_time(counter)}")
A Flask CLI command that creates a Dask Client
to connect to the cluster and execute 10 tests of need_my_time_test
:
@app.cli.command()
def itests(extended):
with Client(processes=False) as dask_client:
futures = dask_client.map(need_my_time_test, range(10))
print(f"Futures: {futures}")
print(f"Gathered: {dask_client.gather(futures)}")
EDIT: For convenience let's add an application factory for an easier reproducible example:
def create_app():
app = Flask(__name__)
app.config.from_mapping(
SECRET_KEY='dev',
DEBUG=True,
)
@app.route('/hello')
def hello():
return 'Hello, World!'
@app.cli.command()
def itests(extended):
with Client(processes=False) as dask_client:
futures = dask_client.map(need_my_time_test, range(10))
print(f"Futures: {futures}")
print(f"Gathered: {dask_client.gather(futures)}")
Using the above with flask itests
, we are running into the following error (described here ):
RuntimeError: Working outside of application context.
This typically means that you attempted to use functionality that needed to interface with the current application object in some way. To solve this, set up an application context with app.app_context().
We have tried:
app_context
( app.app_context().push()
) on the app singleton creation.with current_app.app_context():
on the CLI command and some of the functions that use the current_app
. app_context
through a Dask Variable
but it cannot serialize the context.With no avail.
The questions:
When using current_app
proxy, it is assumed that the Flask app is created in the same process that the proxy is used.
This is not the situation when tasks submitted to the workers are run. The tasks are executed isolated away from the Flask app created in the process that submitted the tasks.
In the task, define the flask app and provide the application context there.
import time
from flask import Flask
from dask.distributed import Client
def _create_app():
app = Flask(__name__)
app.config.from_mapping(
SECRET_KEY='dev',
DEBUG=True,
TAKE_YOUR_TIME=0.2
)
return app
def will_take_my_time(n):
# Add the 'TAKE_YOUR_TIME' in the config in how many seconds you want
app = _create_app()
with app.app_context():
time.sleep(app.config['TAKE_YOUR_TIME'])
return n
def need_my_time_test(counter=None):
print(f"Test No. {will_take_my_time(counter)}")
def create_app():
app = _create_app()
@app.route('/hello')
def hello():
return 'Hello, World!'
@app.cli.command()
def itests():
with Client(processes=False) as dask_client:
futures = dask_client.map(need_my_time_test, range(10))
print(f"Futures: {futures}")
print(f"Gathered: {dask_client.gather(futures)}")
return app
app = create_app()
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.