Python: Transfer a class method to another computer

Question

I have created an class that is used for analysising a specific type of data that I produce. I use this class on a local computer but occasionally there is too much data to work locally so I wanted to add an option to one of methods so that it can submit the job to a computer cluster. It mostly works except I am struggling to transfer a class method to the cluster.

My class looks like this

class Analysis():
    def __init__(self, INPUT_PARAMETERS ETC):
        self.data
        OTHER_STUFF...
    @staticmethod
    def staticMethod1(input1, input2):
        # PERFORM SOME KIND OF CALCULATION ON INPUT1 AND INPUT2 AND RETURN THE RESULT
        return output

    @staticmethod
    def staticMethod2(input1, input2):
        # PERFORM SOME KIND OF CALCULATION ON INPUT1 AND INPUT2 AND RETURN THE RESULT
        return output

    # MORE STATIC METHODS

    @staticmethod
    def staticMethodN(input1, input2):
        # PERFORM SOME KIND OF CALCULATION ON INPUT1 AND INPUT2 AND RETURN THE RESULT
        return output

    def createArray(self, function):
        # CREATE AN ARRAY BY APPLYING FUNCTION TO SELF.DATA
        return array

So the createArray method gets called and the user passes the static method that should be used to calculate the array. When I wanted the array in createArray to be created on the cluster I saved the static method (that was passed to the this method eg staticMethod1 ) into a Pickle file using dill.dump . The Pickle file is then passed to the cluster but when I try to load the method from the Pickle file it says ModuleNotFoundError: No module named 'analysis' which is the module that the Analysis class can be found in.

Do I really need to recreate the whole class on the cluster just to use a static method? Can anyone suggest a elegant fix to this problem or suggest a better way of implementing this functionality? It needs to work with any static method. FYI, one of the static methods uses from sklearn.metrics.cluster import adjusted_rand_score just incase that may effect a solution using dill .

Answer 1

I'm the dill author. dill is able to pass a class method to another computer, as seen below.

>$ python
Python 3.5.6 (default, Sep 20 2018, 12:15:10) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo(object):
...   def bar(self, x):
...     return self.y + x
...   def __init__(self, y):
...     self.y = y
... 
>>> import dill
>>>          
>>> f = Foo(5)
>>>                  
>>> with open('foo.pkl', 'wb') as pkl:
...   dill.dump(f.bar, pkl)
... 
>>>

Then in a new session (or on another computer)...

>$ python
Python 3.5.6 (default, Sep 20 2018, 12:15:10) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> with open('foo.pkl', 'rb') as pkl:
...   b = dill.load(pkl)
... 
>>> b(4)
9

Without more specific code from you, it's hard to say why you aren't seeing this behavior... but dill does provide the ability to pass a class definition (or just a class method) to another computer.

This behavior is what enables code like pathos to pass the class method to another computer within a ParallelPool or a ProcessPool -- the latter is across processes, while the former can be across distributed resources.

dude>$ python
Python 3.5.6 (default, Sep 20 2018, 12:15:10) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> class Foo(object):
...   def bar(self, x):
...     return self.y + x
...   def __init__(self, y):
...     self.y = y
... 
>>> import pathos
>>> p = pathos.pools.ParallelPool()
>>> p.map(Foo(4).bar, [1,2,3])
[5, 6, 7]
>>> p.close(); p.join()
>>>