Varnish doesn't cache without expire header

Question

On my server I have Varnish (caching) running on port 80 with Apache on 8080.
Varnish caches very well when I set the headers like below:

    $this->getResponse()->setHeader('Expires', '', true);
    $this->getResponse()->setHeader('Cache-Control', 'public', true);
    $this->getResponse()->setHeader('Cache-Control', 'max-age=2592000');
    $this->getResponse()->setHeader('Pragma', '', true);

But this means people cache my website without ever retrieving a new version when its available.
When I remove the headers people retrieve a new version every page reload (so Varnish never caches).

I can not figure out what goes wrong here.
My ideal situation is people don't cache the html on the client side but leave that up to Varnish.

Answer 1

My ideal situation is people don't cache the html on the client side but leave that up to Varnish.

What you want is varnish to cache the resource and serve it to clients, and only generate a new version if something changed. The easiest way to do this is have varnish cache it for a long time, and invalidate the entry in varnish (with a PURGE command) when this something changed.

By default, varnish will base its cache rules on the headers the back-end supplies. So, if your php code generates the headers you described, the default varnish vcl will adjust its caching strategy accordingly. However, it can only do this in generalized, safe way (eg if you use a cookie, it will never cache). You know how your back-end works, and you should change the cache behavior of varnish not by sending different headers from the back-end, but write a varnish .vcl file. You should tell varnish to cache the resource for a long time even though the Cache-Control of Max-Age headers are missing (set the TimeToLive ttl in your .vcl file). Varnish will then serve the generated entry until ttl has passed or you purged the entry.

If you've got this working, there's a more advanced option: cache the resource on the client but have the client 'revalidate' it every time it want to use it. A browser does this with an HTTP GET plus If-Modified-Since header (your response should include a Date header to provoke his behavior) or If-Match header (your response should include an ETAG header to provoke his behavior). This saves bandwith because varnish can respond with a 304 NOT-MODIFIED response, without sending the whole resource again.

Answer 2

Simplest approach is just to turn down the max-age to something more reasonable. Currently, you have it set to 30 days. Try setting it to 15 minutes:

$this->getResponse()->setHeader('Cache-Control', 'max-age=900');

Web caching is a somewhat complicated topic, exacerbated by some very different client interpretations. But in general this will lighten the load on your web server while ensuring that new content is available in a reasonable timeframe.

Answer 3

Set your standard HTTP headers for the client cache to whatever you want. Set a custom header that only Varnish will see, such as X-Varnish-TTL Then in your VCL, incorporate the following code in your vcl_fetch sub:

if (beresp.http.X-Varnish-TTL) {
    C{
        char *ttl;
        /* first char in third param is length of header plus colon in octal */
        ttl = VRT_GetHdr(sp, HDR_BERESP, "\016X-Varnish-TTL:");
        VRT_l_beresp_ttl(sp, atoi(ttl));
    }C
    unset beresp.http.X-Varnish-TTL;  // Remove so client never sees this
}