将任务提交到Dask分布式集群时，本地python文件导入问题

Question

I have the following issue. If my code is separated into different files, I cannot simply import and submit function to a cluster, because my local files are not present on workers machines. In order to tackle the problem, I need to manually merge files into one. It's can be done relatively simply for small examples but with a large project that is separated into several folders is very time-consuming. Example:

localfile.py file:

def custom():
  print("Remote run")
  return

Now let's try to submit imported function to a cluster, cluster.py file:

import localfile

x = client.submit(localfile.custom)

x.result()
# Import error here, details below

-

# The protocol we write by default.  May be less than HIGHEST_PROTOCOL.
  File "/usr/local/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 944, in subimport
ModuleNotFoundError: No module named 'localfile'

As you can see workers are not able to locate local python files so the task is failed.

Is there is any way I can send my pickled code with imported local files? I understand that possible solution is just to upload local files to the workers, but when a code is modified very often and your cluster is dynamic is not an elegant solution. It would be great to be able to merge automatically your 'client.py' code with 'localfile.py', pickle it and send to the scheduler. Is it somehow possible? Do you have any other ideas how to solve this problem? Thank you!

Answer 1

As I think you will know, the client API provides upload_file , which can handle getting your module to workers into a place that it can be imported.

In the situation that workers come and go, you will not be able to ensure that a new worker has the file in the right place. However, pickle, by its nature, refers objects/functions to source files, presumably in order to reduce the data size.

Two vague possibilities:

• you could perform compute operations specifically only on the same workers ( compute(workers=) ) that are live at the time of the upload, together with reload or other importlib hooks within the function that you run.
• you could actually distribute function source as text, if you wished, and execute the definitions or, again, use importlib to make it into a module on the workers.

In either case, you would need to modify or wrap your function with something to deal with the extra step.