dask jobqueue 动态分配的工人资源

Question

I'm a new user of dask.jobqueue and I'm trying to schedule tasks thanks to it.

The number of tasks to launch can vary (from a few dozen to several hundred) and the graph of dependencies between these tasks can be quite complex. Also, the resource demands of each of these tasks can vary greatly because some will be highly multi-threaded while others will not. However, I don't have the impression that it's possible at the moment to dynamically vary the resources of the "worker".

for instance the code snippet :

@dask.delayed()
def first_task(...)
    """mono thread task
    """
...
@dask.delayed()
def second_task(...)
    """higly multithreaded task
    """
...

cluster = PBSCluster(...)
cluster.scale(...) # or cluster.adapt(...)
client = Client(cluster)

first_return = first_task()# mono thread
second_return = second_task(first_return) # threaded
third_task(second_return) # highly multitraded

# launch tasks
third_task.compute()

The treatments are triggered thanks to .compute() which can take the named parameter 'resources' but from what I understand ( https://distributed.dask.org/en/latest/resources.html ) corresponds to resources previously informed to each worker at the creation of the cluster, which does not correspond to a dynamic management of the resources per worker that I am looking for.

Basically, is it possible to switch from :

third_task.compute(resources={...})

to something more related to each of the tasks to be processed? Attach somehow resources needed for each dask.delayed?

@dask.delayed(resources={"cpu":1, "ram": "5GB"})
def first_task(...)
...
@dask.delayed(resources={"cpu":24, "ram": "120GB"})
def third_task(...)
...
third_task.compute()

Thanks,

Answer 1

截至 2020-03-27 答案是否定的