Python缓存/存储和线程锁定

Question

When asked my application should open a .csv file make a dict from it. I also need to cache that data and return from cache if its 'young' enough. Did that with a decorator:

def memorize(key, period):
    """
    Memorizing decorator. Returning cached data
    if its validity period is not expired
    """
    def _decoration_wrapper(func):
        def _caching_wrapper(*args, **kwargs):
            cache_key = key
            now = time.time()

            if _timestamps.get(cache_key, now) > now:
                return _cache[cache_key]

            ret = func(*args, **kwargs)
            _cache[cache_key] = ret
            _timestamps[cache_key] = now + period
            return ret
        return _caching_wrapper
    return _decoration_wrapper

Works fine but i need to make it thread-safe. I'd prefer to do it with the Lock class. So i can be sure that in one period only one thread, one time will do the wrapped function.

So i want to now how to do that.
Info how to make tests would be very appreciated.
Much thanks.

Answer 1

I would do this:

import threading

#1
lock = threading.Lock() # use this line if theadsafety for _cache is important

def memorize(key, period):
    """
    Memorizing decorator. Returning cached data
    if its validity period is not expired
    """
    #2
    lock = threading.Lock() # use this line if theadsafety for a key is important
    def _decoration_wrapper(func):
        #3
        lock = threading.Lock() # use this line if theadsafety for  the function is important
        def _caching_wrapper(*args, **kwargs):
            cache_key = key
            now = time.time()

            if _timestamps.get(cache_key, now) > now:
                return _cache[cache_key]
            with lock:
                if _timestamps.get(cache_key, now) > now:
                    return _cache[cache_key]
                ret = func(*args, **kwargs)
                _cache[cache_key] = ret
                _timestamps[cache_key] = now + period
                return ret
        return _caching_wrapper
    return _decoration_wrapper

Line # :

1. you have got a shared ressource, the cache. It might be useful to lock this one. It it is only a dictionairy then the GlobalInterpreterLock will do this for you.
2. Lock the key. If two functions have the same decorator then they are not executed in parallel.
3. Lock the function. This means that a function will wait for the other calls to it in other threads to complete before it looks up the cache again.

How do you test this? Write tests first, put in them more locks and timeouts to assure two functions are not executed at the same time. I do not have a general guide for it but these tests can be really ugly.