Why prefect flow crashes with deep recursion error?

Question

Code

vault_client = hvac.Client()
new_password = secrets.token_urlsafe(16)
accounts_list = ['account_1', 'account_2'] # dicts


###
# Prefect Tasks
###

@task()
def read_old_secret(account: dict):
    old_secret = vault_client.secrets.kv.v2.read_secret(path=account['secret_path'])
    return old_secret['password']


@task()
def rotate_password(account: dict, old_password: str, new_password: str):



@task()
def store_new_password_to_vault(account: dict, new_password: str):
    vault_client.secrets.kv.v2.create_or_update_secret(
        mount_point='point',
        path=account['secret_path'],
        secret={"password": new_password}
    )


###
# Prefect Flow definition and task orchestration
###

with Flow(
        name="flow",
        storage=Local(),
        executor=LocalDaskExecutor(scheduler=config.get('core', 'scheduler'),
                                   num_workers=config.getint('core', 'num_workers')),
        result=LocalResult(dir=config.get('core', 'local_result_dir'))
) as flow:
    for account in accounts_list:
        old_password = read_old_secret(account=account)
        rotate_password(account=account, old_password=old_password, new_password=new_password)
        store_new_password_to_vault(account=account, new_password=new_password)

# Run flow locally (for tests)
if __name__ == '__main__':
    flow.run()

While trying to run flow it crashes with error

[2022] INFO - prefect.FlowRunner | Beginning Flow run for 'flow'
[2022] ERROR - prefect.FlowRunner | Unexpected error: PicklingError('Could not pickle object as excessively deep recursion required.')
Traceback (most recent call last):
  File "/data/prefect/venv/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 633, in dump
    return Pickler.dump(self, obj)
  File "/data/prefect/venv/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 709, in reducer_override
    if sys.version_info[:2] < (3, 7) and _is_parametrized_type_hint(obj):  # noqa  # pragma: no branch
RecursionError: maximum recursion depth exceeded in comparison

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/prefect/venv/lib/python3.9/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/data/prefect/venv/lib/python3.9/site-packages/prefect/engine/flow_runner.py", line 643, in get_flow_run_state
    final_states = executor.wait(
  File "/data/prefect/venv/lib/python3.9/site-packages/prefect/executors/dask.py", line 685, in wait
    return dask.compute(
  File "/data/prefect/venv/lib/python3.9/site-packages/dask/base.py", line 603, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/data/prefect/venv/lib/python3.9/site-packages/dask/multiprocessing.py", line 233, in get
    result = get_async(
  File "/data/prefect/venv/lib/python3.9/site-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "/data/prefect/venv/lib/python3.9/site-packages/dask/local.py", line 319, in reraise
    raise exc
  File "/data/prefect/venv/lib/python3.9/site-packages/dask/multiprocessing.py", line 118, in pack_exception
    result = dumps((e, tb))
  File "/data/prefect/venv/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/data/prefect/venv/lib/python3.9/site-packages/cloudpickle/cloudpickle_fast.py", line 640, in dump
    raise pickle.PicklingError(msg) from e
_pickle.PicklingError: Could not pickle object as excessively deep recursion required.
[2022] ERROR - prefect.flow | Unexpected error occured in FlowRunner: PicklingError('Could not pickle object as excessively deep recursion required.')

Tried already sys.setrecursionlimit(3000), didn't help When i run this code locally and without prefect methods - it works without any errors, Flow goes on prefect server, when i comment vault's operations(hvac);didn.t find in search how to debug deep recursion at all.., Please. help.

Answer 1

You are using LocalDaskExecutor processes which will pickle anything needed by the functions when they are handed to the processes.

Hard to say but if I have to guess, hvac.Client() is not serializable. You can test this by doing:

import cloudpickle
cloudpickle.dumps(hvac.Client())

and I suspect the error will reproduce. You can avoid this by creating the Client inside the task.