Python: HTTP caching using 'CacheControl` not working

Question

I'm using python 3.6 with requests module for API consumption and CacheControl module for caching the API response. I'm using following code but cache does not seems to be working:

import requests
from cachecontrol import CacheControl

sess = requests.session()
cached_sess = CacheControl(sess)

response = cached_sess.get('https://jsonplaceholder.typicode.com/users')

Every request to this URL returns the 200 status code (instead of 304 status code) and the same resource is requested each time even though the ETag headers is same and max-age was still valid. The API returns following cache related headers:

'Cache-Control': 'public, max-age=14400'
'Expires': 'Sat, 04 Feb 2017 22:23:28 GMT' (time of original request)
'Etag': 'W/"160d-MxiAGkI3ZBrjm0xiEDfwqw"'

What could be the issue here?

UPDATE: I'm not sending If-None-Match header with any API call, do I manually have to do it or CacheControl module should take care of it automatically?

Answer 1

Use a cache implementation to persist the cache between runs of the program.

from cachecontrol.caches import FileCache

sess = requests.session()
cached_sess = CacheControl(sess, cache = FileCache('.web_cache'))

Also, ensure you're using a recent CacheControl release. CacheControl has only cached resources served as Transfer-Encoding: chunked since 0.11.7:

$ curl -si https://jsonplaceholder.typicode.com/users | fgrep -i transfer-encoding
Transfer-Encoding: chunked

Every request to this URL returns the 200 status code

This is what you'll see when CacheControl is working correctly. The return of a cached response, or use of a 304, is hidden from you as a client of the code. If you believe that a fresh request is being made to the upstream server, consider something like:

import logging

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)

to see what cachecontrol.controller and requests.packages.urllib3.connectionpool are doing.