Django-redis-cache failing to fetch data from redis

Question

I have a question regarding the working of this package. How does it writes to redis db ?

This is my settings for redis -

CACHES = {
     'default': {
    'BACKEND': 'redis_cache.RedisCache',
    'LOCATION': '/var/run/redis/redis.sock',
    'OPTIONS': {
        'DB': 2,
    },
},
}

This is my views file,

def postview(request):
    print("Working")
   #post_list = Post.objects.all()
    if cache.get("posts") == None:
            post_list = Post.objects.all()
            print("going to be cached")
            a = cache.get("aman")
            print("aman ", a)
            cache.set("posts", post_list, timeout=60*100*10)
            print("cached")
    else :
            post_list = cache.get("posts")
            aman = cache.get("aman")
            print(aman, " aman's job")
            print("already present in cache")
    context = {"post_list" : post_list}
    print("Problem")
    return render(request, 'post_list.html', context)

@cache_page(60*15*10, key_prefix="cache_redis")
def testview(request):
    post_list = cache.get("posts")
    print("post_list is", post_list)
    return render(request, 'post_list.html', {"post_list":post_list})

@cache_page(60*25*10, key_prefix="cache_test")
def new(request):
    print("Hey")
    print("cache_page is working")
    return HttpResponse("Hello, I am Mohammed")

This is my redis -cli ,

luvpreet@DHARI-Inspiron-3542:~/test_venv_wrapper/test_redis/cache_redis$ redis-cli
127.0.0.1:6379> select 2
 OK
127.0.0.1:6379[2]> set "a" "aman"
 OK
127.0.0.1:6379[2]> set ":1:a" "theman"
 OK
 127.0.0.1:6379[2]> keys *
 1) "a"
 2)":1:views.decorators.cache.cache_page..GET.ad00468064711919773512f81be0dbc4.d41d8cd98f00b204e9800998ecf8427e.en-us.UTC"
 3) ":1:posts"
 4) ":1:a"
 5) ":1:aman"
 6) ":1:views.decorators.cache.cache_header.cache_test.ad00468064711919773512f81be0dbc4.en-us.UTC"
 7) ":1:views.decorators.cache.cache_header..ad00468064711919773512f81be0dbc4.en-us.UTC"
 8) ":1:b"
 9)":1:views.decorators.cache.cache_page.cache_test.GET.ad00468064711919773512f81be0dbc4.d41d8cd98f00b204e9800998ecf8427e.en-us.UTC"
10) "aman"
127.0.0.1:6379[2]> get ":1:a"
"theman"
127.0.0.1:6379[2]> get "a"
"aman"

This is the corresponding redis-cli monitor

luvpreet@DHARI-Inspiron-3542:~/test_venv_wrapper/test_redis/cache_redis$ redis-cli monitor
OK
1491412249.001149 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412249.086196 [0 127.0.0.1:44984] "select" "2"
1491412250.001249 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412257.001426 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412257.423536 [2 127.0.0.1:44984] "set" "a" "aman"
1491412258.001311 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412269.001211 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412269.820886 [2 127.0.0.1:44984] "set" ":1:a" "theman"
1491412270.000741 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412272.955386 [2 127.0.0.1:44984] "keys" "*"
1491412273.001121 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412340.991928 [2 127.0.0.1:44984] "get" ":1:a"
1491412341.002001 [0 unix:/var/run/redis/redis.sock] "INFO"
1491412344.106985 [2 127.0.0.1:44984] "get" "a"
1491412345.001677 [0 unix:/var/run/redis/redis.sock] "INFO"

This means that data which I am manually inserting into the database 2 is available and I can fetch that through redis-cli.

But when I try to fetch this manually entered data from the django-app python shell, This happens,

>>> from django.core.cache import cache
>>> cache.get("a")
Traceback (most recent call last):
   File "<console>", line 1, in <module>
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 33, in wrapped
     return method(self, client, key, *args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 259, in get
     value = self.get_value(value)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 210, in get_value
     value = self.deserialize(value)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/backends/base.py", line 197, in deserialize
     return self.serializer.deserialize(value)
   File "/usr/local/lib/python2.7/dist-packages/redis_cache/serializers.py", line 42, in deserialize
     return pickle.loads(force_bytes(value))
UnpicklingError: could not find MARK

Here is the redis-cli monitor corresponding to it,

OK
1491413058.004167 [0 unix:/var/run/redis/redis.sock] "INFO"
1491413059.002746 [0 unix:/var/run/redis/redis.sock] "INFO"
1491413060.663292 [2 unix:/var/run/redis/redis.sock] "GET" ":1:a"
1491413061.001167 [0 unix:/var/run/redis/redis.sock] "INFO"

Why cannot I access the data written manually ? I know that it adds a prefix to the data written through django. The prefix is ":1:key_name" . That's why I added 2 keys namely "a" and ":1:a". So that when I try to access the "a", it will call ":1:a".

But this error comes up. So, it definitely might be writing data to redis by some other way. Please tell me about the error and also tell me the way in which it writes data.

Answer 1

The error you are gettting is not a data retreival error, the data is being retreived, but it is not in correct pickle format.

• When you are setting data manually, it is being set in byte format, not pickled format.

• When you set data through django-redis, your data is first serialized using cPickle, and then stored into redis as pickled string.

• When you retreive the data stored through django-redis, it is retreived as the pickle serialized string, and then deserialized into corresponding python datatype.

• Your manually entered data is string type, but not in correct pickle format, hence though it gets retreived, it cannot be converted into corresponding python type, and you get the pickling error.
• The solution is to use django-redis itself in the django shell or your code to store and retreive data, entering data manually breaks the serialization contract.

Example:

>>> from django.core.cache import cache
>>> cache.set("THE_KEY","aman")
True
>>> cache.get("THE_KEY")
'aman'
>>> 
dhruv@dhruvpathak:~$ redis-cli
127.0.0.1:6379> keys *
 1) ":1:THE_KEY"
127.0.0.1:6379> get ":1:THE_KEY"
"\x80\x02U\x04amanq\x01."
127.0.0.1:6379> set "THE_MANUAL_KEY" "aman"
OK
127.0.0.1:6379> get "THE_MANUAL_KEY" 
"aman"