Nginx + php5-fpm + varnish + APC上的Wordpress高CPU和内存使用率

Question

After days of debugging and tweaking around with settings, I'm getting exhausted & unable to find a solution. Kindly guide.

I've the following server on DigitalOcean:

64GB Memory
8 Core processor
200GB SSD drive

And I'm running a single Wordpress site on it. Site gets high traffic. (2000 to 3000 concurrent realtime users) And I'm sure due to my bad settings I'm losing traffic & unable to serve pages to users. I expect the realtime users to be 5000+ but it always stays around 2000.

I constantly get OOM errors and due to which mysql or php5-fpm gets killed and the site goes down. If I tweak php-fpm and nginx I get 502 and 503 errors. Or I get upstream timed out (110: Connection timed out)' or FastCGI sent in stderr: PHP message: PHP Fatal error: Maximum execution time of 30 seconds exceeded error.

Now, I've tweaked the settings so that I don't get any error but the traffic has dropped to around 1500 concurrent users and it refuses to go up. So I'm sure there's something wrong in my settings.

/etc/php5/fpm/pool.d/www.conf settings:

pm = dynamic
pm.max_children = 150
pm.start_servers = 40
pm.min_spare_servers = 30
pm.max_spare_servers = 50
pm.max_requests = 1000

FastCGI settings: /etc/nginx/conf.d/default.conf

location ~ \.php$ {
             try_files $uri =404;
             # proxy buffers - no 502 errors!
             proxy_buffer_size               128k;
             proxy_buffers                   4 256k;
             proxy_busy_buffers_size         256k;

            fastcgi_buffers 256 16k;
            fastcgi_buffer_size 128k;
            fastcgi_max_temp_file_size 0;
            fastcgi_intercept_errors on;
            fastcgi_keep_conn off;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME    $document_root$fastcgi_script_name;
            fastcgi_pass unix:/dev/shm/php-fpm-www.sock;


        }

APC setting: /etc/php5/fpm/php.ini

[apc]
apc.write_lock = 1
apc.slam_defense = 0
apc.shm_size = "1024M"

I've noticed that php5-fpm processes take a lot of memory. Eg when I calculate the average memory per process I get: ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d%s\\n", sum/NR/1024,"M") }' ps --no-headers -o "rss,cmd" -C php5-fpm | awk '{ sum+=$1 } END { printf ("%d%s\\n", sum/NR/1024,"M") }' gives me 238M for a concurrent traffic of 1100.

Please guide me where my config is incorrect. Because I'm 100% sure my traffic is choking.

---

Additional info

Nginx config: /etc/nginx/nginx.conf

worker_processes  24;
worker_rlimit_nofile 20000;


events {
    worker_connections  40000;
    use epoll;
    multi_accept on;
}

But I've noticed that ulimit on the server is:

ulimit -n shows 1024 only. Is this related to my issue?

---

Added VCL after Daniel's response

# This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.
#
# Default backend definition.  Set this to point to your content
# server.
#
backend default {
    .host = "127.0.0.1";
    .port = "8080";
        .connect_timeout = 600s;
        .first_byte_timeout = 600s;
        .between_bytes_timeout = 600s;
        .max_connections = 800;
}


acl purge {
        "localhost";
}

sub vcl_recv {
        set req.grace = 6h;

  # Set X-Forwarded-For header for logging in nginx
  remove req.http.X-Forwarded-For;
  set    req.http.X-Forwarded-For = client.ip;


  # Remove has_js and CloudFlare/Google Analytics __* cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
  # Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");



# Either the admin pages or the login
if (req.url ~ "/wp-(login|admin|cron)") {
        # Don't cache, pass to backend
        return (pass);
}

# Remove the wp-settings-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");

# Remove the wp-settings-time-1 cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");

# Remove the wp test cookie
set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");

# Static content unique to the theme can be cached (so no user uploaded images)
# The reason I don't take the wp-content/uploads is because of cache size on bigger blogs
# that would fill up with all those files getting pushed into cache
if (req.url ~ "wp-content/themes/" && req.url ~ "\.(css|js|png|gif|jp(e)?g)") {
    unset req.http.cookie;
}

# Even if no cookies are present, I don't want my "uploads" to be cached due to their potential size
if (req.url ~ "/wp-content/uploads/") {
    return (pass);
}

# Check the cookies for wordpress-specific items
if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
        # A wordpress specific cookie has been set
    return (pass);
}



        # allow PURGE from localhost
        if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                return (lookup);
        }


        # Force lookup if the request is a no-cache request from the client
        if (req.http.Cache-Control ~ "no-cache") {
                return (pass);
        }
# Try a cache-lookup
return (lookup);

}

sub vcl_fetch {
        #set obj.grace = 5m;
    set beresp.grace = 6h;

}

sub vcl_hit {
        if (req.request == "PURGE") {
                purge;
                error 200 "Purged.";
        }
}

sub vcl_miss {
        if (req.request == "PURGE") {
                purge;
                error 200 "Purged.";
        }
}

Answer 1

With that traffic and this config, you're missing a lot of opportunities (and income) :). You really should get your hands on Nginx microcaching and Varnish!