Cannot unregister functions from atexit in python 2.7

Question

First, I wrote a recording class with a flush method:

class Recorder
    def __init__(self, buffer_size, path):
        self._big_buffer = np.array(*buffer_size)
        self._path = path
    def push(self, data):
        # insert in self._big_buffer
        # if self._big_buffer is full:
        #     self._flush()
    def flush(self):
        # write buffer to disk (self._path)

Then, I wanted to flush at exit: when manually stopped, crashed or whatever reason.

So I used:

def __init__(self):
    (...)
    atexit.register(self.flush)

And it worked pretty well.

But now, I want to record, stop recording, record again, multiple times, with a different buffer size and to a different path. So I have to discard, then instanciate several Recorder . It kind of works, but older Recorder 's memory (containing some fat self._big_buffer̀ ) is not freed since it's retained by atexit . Even when I explicitly call del . I can't atexit.unregister(self._flush) since it's Python 3 only.

I would prefer not to reuse existing instances, but discarding older instances and create new ones.

How would you handle such a case?

Answer 1

You can try using a weak reference to the atexit handler, so the object won't be retained if it is deleted elsewhere:

import atexit
import weakref

class CallableMethodWeakRef:
    def __init__(self, object, method_name):
        self.object_ref = weakref.ref(object)
        self.method_name = method_name
    def __call__(self):
        object = self.object_ref()
        if object:
            getattr(object, self.method_name)()

class Recorder:
    def __init__(self, *args):
        atexit.register(CallableMethodWeakRef(self, 'flush'))

    def flush(self):
        print 'flushing'

The method is passed as a string in order to avoid a lot of problems with bound method weak references, if you find it disturbing you can always use a BoundMethodWeakref implementation like this one: http://code.activestate.com/recipes/578298-bound-method-weakref/

Answer 2

I would say you're trying to use the wrong tool. The with statement and context managers are a very good tool for this. File IO is the main example that most python users will get introduced to the with statement.

f = open("somefile.txt", "w")
try:
    f.write("...")
    # more file operations
finally:
    # regardless of what happens, make sure the files is closed 
    f.close()

Becomes:

with open("somefile.txt", "w") as f:
    f.write("...")
    # more file operations
# close automatically called at the end of the block

You can create your own context managers by writing __enter__ and __exit__ methods for your class.

class Recorder
    def __init__(self, buffer_size, path):
        self._big_buffer = np.array(*buffer_size)
        self._path = path
    def push(self, data):
        # insert in self._big_buffer
        # if self._big_buffer is full:
        #     self._flush()
    def flush(self):
        # write buffer to disk (self._path)
    def __enter__(self):
        return self
    def __exit__(self, exctype, exception, traceback):
        # If an exception was thrown in the with block you will get the details here. 
        # If you want the say that the exception has been handled and for it not to be 
        # raised outside the with block then return True
        self.flush()
        # self.close() ?

You would then use your Recorder object like:

with Recorder(...) as recorder:
    # operations with recorder
    ...
# regardless of what happens the recorder will be flushed at this point

Answer 3

Surely the answer is to allow your Recorder to change paths and buffer characteristics at will. You say "I would prefer not to reuse existing instances, but discarding older instances and create new ones." but you don't give any rationale for that, except perhaps your assumption that the "older Recorder's memory (containing some fat self._big_buffer̀ ) is not freed since it's retained by atexit " , which I believe is incorrect.

While it is true that atexit retains a reference to the recorder object, this will only mean that the buffer memory is retained as long as the recorder refers to it. It would be quite easy to add a close() method such as

    def close(self):
        self.flush()
        self._big_buffer = None

and bingo! No reference to the buffer memory exists, and it is collectable.

Your __init__() method should simply register with atexit , then the open() method (which does the rest of what __init__() currently does) can be used multiple times, each one followed by a close() call.

In summary, I think your problem cries out for a single object.

Answer 4

You can remove the handle by hand from the (undocumented) atexit._exithandlers list.

import atexit

def unregister(func, *targs, **kargs):

    """unregister a function previously registered with atexit.
       use exactly the same aguments used for before register.
    """
    for i in range(0,len(atexit._exithandlers)):
        if (func, targs, kargs) == atexit._exithandlers[i] :
            del atexit._exithandlers[i]
            return True
    return False

Hope that helps.