Varnish / Apache Random 503错误

Question

I'm running cPanel on a Rackspace cloud server that's hosting about 30 websites. I'm using Varnish 3.0.3 + Apache 2.2.23. I'm getting random 503 errors. If I restart the httpd process, they go away. I checked the Apache error_log, and I don't see anything related to a 503 error.

Here's my Varnish config:

backend default {
    .host = "198.61.161.65";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    }

Here's my Varnish startup parameters:

-a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -p http_max_hdr=256 -p http_resp_hdr_len=8192 -p http_range_support=on -s malloc,768M

I ran varnishlog and recorded some of the errors. They all have TxResponse=Service Unavailable in them. To me, this means Apache isn't responding to the request from Varnish in a timely manner. Here's an example:

17 SessionOpen  c 74.133.75.136 54227 :80
17 ReqStart     c 74.133.75.136 54227 1219297893
17 RxRequest    c GET
17 RxURL        c /
17 RxProtocol   c HTTP/1.1
17 RxHeader     c Host: www.archerytrade.org
17 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:20.0) Gecko/20100101 Firefox/20.0
17 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
17 RxHeader     c Accept-Language: en-US,en;q=0.5
17 RxHeader     c Accept-Encoding: gzip, deflate
17 RxHeader     c DNT: 1
17 RxHeader     c Connection: keep-alive
17 VCL_call     c recv lookup
17 VCL_call     c hash
17 Hash         c /   
17 Hash         c www.archerytrade.org
17 VCL_return   c hash
17 VCL_call     c miss fetch
17 Backend      c 18 default default
17 FetchError   c http first read error: -1 0 (Success)
17 VCL_call     c error deliver
17 VCL_call     c deliver deliver
17 TxProtocol   c HTTP/1.1
17 TxStatus     c 503
17 TxResponse   c Service Unavailable
17 TxHeader     c Server: Varnish
17 TxHeader     c Content-Type: text/html; charset=utf-8
17 TxHeader     c Retry-After: 5
17 TxHeader     c Content-Length: 441
17 TxHeader     c Accept-Ranges: bytes
17 TxHeader     c Date: Wed, 17 Apr 2013 01:39:52 GMT
17 TxHeader     c X-Varnish: 1219297893
17 TxHeader     c Age: 0
17 TxHeader     c Via: 1.1 varnish
17 TxHeader     c Connection: close
17 Length      c 441
17 ReqEnd      c 1219297893 1366162792.398471832 1366162792.541639328 0.000244379 0.143036604 0.000130892
17 SessionClose c error
17 StatSess     c 74.133.75.136 54227 0 1 1 0 0 0 257 441

I've tried adding the following to my default.vcl file:

sub vcl_recv {
set req.grace = 15s;
...
}

sub vcl_fetch {
if (beresp.status == 500 || beresp.status == 503) {
set beresp.saintmode = 10s;
return(restart);
} 
set beresp.grace = 1h;
...
}

Around the time of some 503 errors, I found PHP Fatal error, Premature end of script headers, and Request exceeded the limit of 10 internal redirects due to probable configuration error, in the Apache error_log.

I've tried increasing/decreasing the memory for Varnish and the timeout values. I've tried adding the higher http_max_hdr and http_resp_hdr_len values.

Answer 1

We're using APC on this server, and the allotted memory size was only 96M. After increasing the memory a few times and tracking the usage, we settled on 312M. The cache hit rate is 99.7% and the fragmentation is < 0.2%. The Apache 503 errors have almost completely stopped.

Answer 2

Following is what helped me fixing the issue:

After executing following command, I knew about the error:

$ varnishlog -b -q 'FetchError'

Error was "- FetchError http first read error: EOF", details can be found at varnish site

Then after setting up "first_byte_timeout", the issue was fixed:

backend default {
  .host = "127.0.0.1";
  .port = "8888";
  .connect_timeout = 1s; # Wait a maximum of 1s for backend connection (Apache, Nginx, etc...)
  .first_byte_timeout = 5s; # Wait a maximum of 5s for the first byte to come from your backend
  .between_bytes_timeout = 2s; # Wait a maximum of 2s between each bytes sent
}

Details at Varnish 503 Solutions