Share python modules across computers in a dask.distributed cluster

Question

I have a ssh dask.distributed cluster with a main computer containing all modules for my script and another one with only a few, including dask itself of course.

Is it possible to change the sys path of the other computer so that it also looks for modules in the main one? Of course, I could simply upload them via sftp but since I keep making a lot of modules that would be very annoying to do repeatedly.

Answer 1

Such things are possible with networking solutions such as NFS or SSH remote mounts, but that's a pretty big field and beyond the scope of Dask itself. If you are lucky, other answers will appear here, others have solved similar problems, but more likely copying is the simpler solution.