Can Varnish pre-generate pages?

Question

I have a page that takes about 1-2minutes to generate, and needs to be as up to date/fast as possible. The page will be accessed by around 1000 people daily. It's 100% vital that all these users get instant pages and can't be stuck waiting 1-2minutes for a page to load.

The varnish expire time is set to 1 minute. So if a user lands on the page at the end of the minute they will be stuck waiting for it to re-generate. The solution I see is to write a minutely cron to grab the page so it is always up to date... but there will always be a 1-5% leak of users hitting the 1-2minute page load which is unacceptable for my use-case.

Is there a way to ask Varnish to do this in the background? or another work-around?

Thank you.

Answer 1

There is a few optional settings for backend server timeout. See

backend default {
    .host = "127.0.0.1";
    .port = "80";
    .connect_timeout = 2s;
    .first_byte_timeout = 10s; 
    .between_bytes_timeout = 1s;
}

• .connect_timeout - waiting time for server response (ping)
• .first_byte_timeout - waiting time for first byte send from backend server (for PHP backend server, it would be set as max_execution_time)
• .between_bytes_timeout - this is useful when short lost connection

For more information see https://www.varnish-software.com

Answer 2

You can do a "dirty" workaround playing with backends, grace periods and custom headers.

More or less:

1. Change your cron script to send a custom HTTP header (by example "X-Cron: true") (if using curl, -H option)
2. Define a new backend "sick" with fake data and a probe that marks the backend always as sick
3. Set a grace period on vcl_recv of at least 2 minutes
4. Check on vcl_recv if the custom header is present, if not change the backend to the sick one.
5. Set a grace period on vcl_fetch of at least 2 minutes.

With that in place, only cron requests will hit the backend... and the rest will be served from cache.

Be carefull, this "solution" doesn't take care of backend failures

Answer 3

Here I tried to create a configuration to copy and paste using the header suggested by @NITEMAN, I've made the grace period with max 1h for safety and limited the cron check on the generator URL only to prevent the whole site going into grace mode for any reason.

PS: I'm still new to varnish my self, so if there's any problem please help me improve this answer.

I've used the evil hack example from the varnish book with few changes, linked for reference.

backend good {
.host = "localhost"
.port = "8080"; # if this is your nginx port
}
backend bad {
.host "localhost"
.port "12345" #random port
.probe {.url = "/unavailable-url"; .initial = 0; .interval = 1d;}
}
sub vlc_recv {
  if ( req.url ~ '^/generator-url' ) {
    if ( req.http.X-cron == true ) {
      set req.backend = good;
    else {
      set req.backend = bad;
    }
    if ( req.backend.healthy ) {
      set req.grace = 30s;
    } else {
      set req.grace = 1h;
    }
}
sub vcl_fetch {
  set beresp.grace = 1h;
}