Why does this function not work when used as a decorator?

Question

UPDATE : As noted by Mr. Fooz, the functional version of the wrapper has a bug, so I reverted to the original class implementation. I've put the code up on GitHub:

https://github.com/nofatclips/timeout/commits/master

There are two commits, one working (using the "import" workaround) the second one broken.

The source of the problem seems to be the pickle#dumps function, which just spits out an identifier when called on an function. By the time I call Process , that identifier points to the decorated version of the function, rather than the original one.

---

ORIGINAL MESSAGE :

I was trying to write a function decorator to wrap a long task in a Process that would be killed if a timeout expires. I came up with this (working but not elegant) version:

from multiprocessing import Process
from threading import Timer
from functools import partial
from sys import stdout

def safeExecution(function, timeout):

    thread = None

    def _break():
        #stdout.flush()
        #print (thread)
        thread.terminate()

    def start(*kw):
        timer = Timer(timeout, _break)
        timer.start()
        thread = Process(target=function, args=kw)
        ret = thread.start() # TODO: capture return value
        thread.join()
        timer.cancel()
        return ret

    return start

def settimeout(timeout):
    return partial(safeExecution, timeout=timeout)

#@settimeout(1)
def calculatePrimes(maxPrimes):
    primes = []

    for i in range(2, maxPrimes):

        prime = True
        for prime in primes:
            if (i % prime == 0):
                prime = False
                break

        if (prime):
            primes.append(i)
            print ("Found prime: %s" % i)

if __name__ == '__main__':
    print (calculatePrimes)
    a = settimeout(1)
    calculatePrime = a(calculatePrimes)
    calculatePrime(24000)

As you can see, I commented out the decorator and assigned the modified version of calculatePrimes to calculatePrime. If I tried to reassign it to the same variable, I'd get a "Can't pickle : attribute lookup builtins.function failed" error when trying to call the decorated version.

Anybody has any idea of what is happening under the hood? Is the original function being turned into something different when I assign the decorated version to the identifier referencing it?

UPDATE : To reproduce the error, I just change the main part to

if __name__ == '__main__':
    print (calculatePrimes)
    a = settimeout(1)
    calculatePrimes = a(calculatePrimes)
    calculatePrimes(24000)
    #sleep(2)

which yields:

 Traceback (most recent call last): File "c:\\Users\\mm\\Desktop\\ING.SW\\python\\thread2.py", line 49, in <module> calculatePrimes(24000) File "c:\\Users\\mm\\Desktop\\ING.SW\\python\\thread2.py", line 19, in start ret = thread.start() File "C:\\Python33\\lib\\multiprocessing\\process.py", line 111, in start self._popen = Popen(self) File "C:\\Python33\\lib\\multiprocessing\\forking.py", line 241, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\\Python33\\lib\\multiprocessing\\forking.py", line 160, in dump ForkingPickler(file, protocol).dump(obj) _pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtin s.function failed Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\\Python33\\lib\\multiprocessing\\forking.py", line 344, in main self = load(from_parent) EOFError 

PS I also wrote a class version of safeExecution, which has exactly the same behaviour.

Answer 1

Move the function to a module that's imported by your script.

Functions are only picklable in python if they're defined at the top level of a module. Ones defined in scripts are not picklable by default. Module-based functions are pickled as two strings: the name of the module, and the name of the function. They're unpickled by dynamically importing the module then looking up the function object by name (hence the restriction on top-level-only functions).

It's possible to extend the pickle handlers to support semi-generic function and lambda pickling, but doing so can be tricky. In particular, it can be difficult to reconstruct the full namespace tree if you want to properly handle things like decorators and nested functions. If you want to do this, it's best to use Python 2.7 or later or Python 3.3 or later (earlier versions have a bug in the dispatcher of cPickle and pickle that's unpleasant to work around).

Is there an easy way to pickle a python function (or otherwise serialize its code)?

Python: pickling nested functions

http://bugs.python.org/issue7689

EDIT :

At least in Python 2.6, the pickling works fine for me if the script only contains the if __name__ block, the script imports calculatePrimes and settimeout from a module, and if the inner start function's name is monkey-patched:

def safeExecution(function, timeout):
    ...    
    def start(*kw):
        ...

    start.__name__ = function.__name__ # ADD THIS LINE

    return start

There's a second problem that's related to Python's variable scoping rules. The assignment to the thread variable inside start creates a shadow variable whose scope is limited to one evaluation of the start function. It does not assign to the thread variable found in the enclosing scope. You can't use the global keyword to override the scope because you want and intermediate scope and Python only has full support for manipulating the local-most and global-most scopes, not any intermediate ones. You can overcome this problem by placing the thread object in a container that's housed in the intermediate scope. Here's how:

def safeExecution(function, timeout):
    thread_holder = []  # MAKE IT A CONTAINER

    def _break():
        #stdout.flush()
        #print (thread)
        thread_holder[0].terminate() # REACH INTO THE CONTAINER

    def start(*kw):
        ...
        thread = Process(target=function, args=kw)
        thread_holder.append(thread) # MUTATE THE CONTAINER
        ...

    start.__name__ = function.__name__ # MAKES THE PICKLING WORK

    return start

Answer 2

Not sure really why you get that problem, but to answer your title question: Why does the decorator not work?

When you pass arguments to a decorator, you need to structure the code slightly different. Essentially you have to implement the decorator as a class with an __init__ and an __call__ .

In the init, you collect the arguments that you send to the decorator, and in the call, you'll get the function you decorate:

class settimeout(object):
    def __init__(self, timeout):
        self.timeout = timeout

    def __call__(self, func):
        def wrapped_func(n):
            func(n, self.timeout)
        return wrapped_func

@settimeout(1)
def func(n, timeout):
    print "Func is called with", n, 'and', timeout

func(24000)

This should get you going on the decorator front at least.