如何从普通函数中向 Python Dask 调度程序提交任务

Question

I want to do something like this:

client = Client()
def sub():
    client.submit(lambda: 'ok')

Calling sub() does not submit the task. Calling client.submit(lambda: 'ok') directly does. I can't find anything in the docs discribing this behavior. (I am on Dask version 1.1.4 and am currently not able to upgrade)

Answer 1

In your original code, the future generated by submit was not stored in a variable, and there was no reference to it after your function sub completed. Therefore, Dask will have noticed that no client is any more interested in that future, and it will have been cleaned from the system - probably it had already run, and the memory of the result will have been released.

This is expected behaviour, and deciding in your code which futures have references to them is a fundamental concept in managing memory on the system, see https://distributed.dask.org/en/latest/memory.html#clearing-data

I suspect that when yo originally called submit while not in a function, a reference to the result was kept by your repl - if you had added a ; to the end of the line to suppress output, you would have seen the same behaviour as calling sub .

Answer 2

The task was submitted also when I executed sub() but I did not see this reflected in the Dask GUI after refreshing the GUI. You can check for certain that it works if you do

def sub():
    return client.submit(lambda: 'ok')

sub().result()
# 'ok'