Problems with Apache servers and A LOT of httpd processes

Question

We have a problem with our Apache server. It's really slow and cpu usage is really high (sometimes showing a constant 100%).

It's on a ubuntu system.

Any ideas what might be causing this?

"top" shows this (there's probably even more httpd processes that didn't fit in the terminal window):

top - 11:33:19 up  1:07,  2 users,  load average: 37.79, 35.90, 29.14
Tasks:  86 total,  10 running,  76 sleeping,   0 stopped,   0 zombieCpu(s): 90.6%us,  9.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    786432k total,   379584k used,   406848k free,        0k buffersSwap:        0k total,        0k used,        0k free,        0k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                      
 5507 nobody    16   0 54540  19m 5204 S    4  2.6   0:03.70 httpd                          5653 nobody    17   0 59996  24m 4808 R    4  3.2   0:01.98 httpd                         
 3277 nobody    15   0  201m  51m 6724 R    2  6.7   2:14.30 mysqld                         5260 nobody    15   0 53740  18m 4972 S    2  2.4   0:06.53 httpd                         
 5333 nobody    15   0 54536  19m 5152 S    2  2.5   0:06.50 httpd                          3923 nobody    15   0 52644  18m 6372 S    1  2.4   0:16.80 httpd                         
 5519 nobody    15   0 51252  15m 4792 S    1  2.0   0:03.02 httpd                          4048 nobody    15   0 52808  17m 5040 R    1  2.3   0:13.86 httpd                         
 5240 nobody    15   0 53980  18m 4916 S    1  2.4   0:08.42 httpd                         
 5249 nobody    15   0 52908  17m 5188 S    1  2.3   0:08.23 httpd                          5385 nobody    15   0 52568  16m 4824 S    1  2.2   0:04.19 httpd                         
 5395 nobody    16   0 52536  16m 4804 S    1  2.1   0:03.90 httpd                          5422 nobody    16   0 51408  15m 4888 S    1  2.1   0:03.49 httpd                         
 5503 nobody    15   0 52536  16m 4800 S    1  2.1   0:02.67 httpd                          3646 nobody    15   0 52572  16m 5012 S    1  2.2   0:25.42 httpd                         
 3948 nobody    15   0 52844  17m 5008 R    1  2.3   0:17.15 httpd                          5283 nobody    15   0 54000  18m 4944 S    1  2.4   0:07.05 httpd                         
 5300 nobody    15   0 54748  19m 5276 S    1  2.6   0:06.20 httpd                          5304 nobody    15   0 52876  17m 4904 S    1  2.2   0:06.45 httpd                         
 5305 nobody    15   0 52812  17m 4884 S    1  2.2   0:06.02 httpd                          5400 nobody    16   0 52060  16m 4800 R    1  2.1   0:04.10 httpd                         
 5423 nobody    15   0 54764  19m 5196 S    1  2.6   0:04.37 httpd                          5433 nobody    15   0 52168  16m 4844 R    1  2.1   0:03.38 httpd                         
 5497 nobody    15   0 52572  16m 4820 S    1  2.2   0:02.83 httpd                          5498 nobody    15   0 53076  17m 4864 S    1  2.3   0:03.14 httpd                         
 5502 nobody    15   0 51260  15m 4804 S    1  2.0   0:02.70 httpd                          5505 nobody    15   0 54920  19m 4976 S    1  2.5   0:02.68 httpd                         
 5546 nobody    15   0 52584  16m 4664 S    1  2.2   0:02.48 httpd                          5550 nobody    15   0 51244  15m 4796 S    1  2.0   0:02.42 httpd                         
 5652 nobody    15   0 51976  17m 5912 S    1  2.3   0:01.24 httpd                          5659 nobody    15   0 55012  19m 4980 S    1  2.6   0:01.25 httpd                         
 5707 nobody    15   0 51416  15m 4880 S    1  2.1   0:00.09 httpd                          5710 nobody    15   0 51240  15m 4792 S    1  2.0   0:00.14 httpd                         
 5399 nobody    15   0 52568  16m 4856 S    1  2.2   0:03.86 httpd                          5435 nobody    15   0 52596  16m 4864 S    1  2.2   0:02.96 httpd                         
 5467 nobody    15   0 52316  16m 4856 R    1  2.1   0:02.93 httpd                          5501 nobody    15   0 52956  17m 4844 S    1  2.2   0:02.79 httpd                         
 5525 nobody    15   0 54508  19m 4916 S    1  2.5   0:03.41 httpd                          5547 nobody    15   0 52604  18m 5708 S    1  2.3   0:02.77 httpd                         
 5590 root      15   0 19208 1360 1044 R    1  0.2   0:00.39 top                            5627 nobody    15   0 55648  20m 5164 S    1  2.6   0:02.56 httpd                         
 5629 nobody    15   0 51248  15m 4792 S    1  2.0   0:01.62 httpd                          5647 nobody    15   0 52956  17m 4884 S    1  2.2   0:01.16 httpd                         
 5656 nobody    15   0 52588  17m 4884 S    1  2.2   0:00.94 httpd                          5665 nobody    15   0 51244  15m 4792 S    1  2.0   0:01.36 httpd                         
 5709 nobody    15   0 51928  15m 4628 S    1  2.1   0:00.02 httpd                          5711 nobody    16   0 51148  15m 4600 S    1  2.0   0:00.02 httpd                      

Answer 1

Apache needs to fork a process to handle each connected client. Read more about that here .

The upshot is that you have lots of processes because you have lots of connections. If your server is running slowly as a result then here are a few options:

1. Optimise your application: profile your application's code and try and optimise the areas that place the most load on the server.
2. Consider changing to a lower impact web server, such as nginx . This could increase the number of connections that your server can handle.
3. Add page caching, using something like varnish
4. Add another server, and load balance between them
5. Increase the capabilities of the existing server (eg increase RAM, upgrade processor, install a faster hard disk).

All of these options have a cost, in time or money. Unfortunately that's just the way it is :)

The quickest wins in my opinion would probably be options 2 and 3. But it means downtime while they're being installed, or having a backup server to work on.

Answer 2

In my experience, the dozens of httpd processes were DOS attacks. After installing mod_evasive, the dozens of httpd processes showing via top were all but eliminated and my mysql CPU load went from an average 100%+ / 99% of the time utilization down to an expected 20% load during heavier query operations.

To install mod_evasive, on RHEL (CentOS/Fedora/Redhat) systems, do:

yum install -y mod_evasive    

Restart Apache

/sbin/service httpd restart

or

systemctl restart httpd

and viola.

Even if a DOS attack isn't your issue, mitigating attacks before they happen is wise.

Answer 3

System and Httpd Information

System:

 # cat /proc/version Linux version 3.10.0-514.2.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Dec 6 23:06:41 UTC 2016 

Httpd Version:

 # httpd -v Server version: Apache/2.4.6 (CentOS) Server built: Nov 14 2016 18:04:44 

Make MPM Prefork Configure

Get MPM In The Default

 # sudo apachectl -t -D DUMP_MODULES |grep mpm [Thu Jan 12 16:06:11.778080 2017] [so:warn] [pid 23727] AH01574: module wsgi_module is already loaded, skipping mpm_prefork_module (shared) 

That means mpm_prefork_module is installed and enabled. You can change it to the mode of event , worker or prefork(default) in the file 00-mpm.conf , which file path is /etc/httpd/conf.modules.d .

The reference of Apache MPM is Apache MPM Common Directives

Add Personal Configure

# ps aux|grep httpd

root     16576  0.0  0.0 232232   504 ?        Ss   13:40   0:01 /usr/sbin/httpd -DFOREGROUND
apache   24296 10.9  4.8 542936 91500 ?        S    16:29   0:16 /usr/sbin/httpd -DFOREGROUND
apache   24310 10.6  4.7 542804 89752 ?        S    16:29   0:15 /usr/sbin/httpd -DFOREGROUND
apache   24343 11.2  6.5 542804 123944 ?       S    16:29   0:15 /usr/sbin/httpd -DFOREGROUND
apache   24344 11.1  6.5 542804 124084 ?       S    16:29   0:14 /usr/sbin/httpd -DFOREGROUND
apache   24345 11.6  6.7 542804 126616 ?       S    16:29   0:15 /usr/sbin/httpd -DFOREGROUND
apache   24347 11.0  6.6 542804 125764 ?       S    16:29   0:14 /usr/sbin/httpd -DFOREGROUND
apache   24377 12.0  6.7 542804 126396 ?       S    16:29   0:15 /usr/sbin/httpd -DFOREGROUND
apache   24378 12.4  6.7 542804 126448 ?       S    16:29   0:15 /usr/sbin/httpd -DFOREGROUND
apache   24408 12.3  6.6 542804 126032 ?       S    16:29   0:14 /usr/sbin/httpd -DFOREGROUND
apache   24450 18.6  5.4 511560 102520 ?       R    16:30   0:10 /usr/sbin/httpd -DFOREGROUND
apache   24458 19.4  4.7 498476 89548 ?        S    16:30   0:08 /usr/sbin/httpd -DFOREGROUND
apache   24466 21.4  4.5 495424 86416 ?        R    16:31   0:07 /usr/sbin/httpd -DFOREGROUND
apache   24484  0.5  0.3 236456  5760 ?        S    16:31   0:00 /usr/sbin/httpd -DFOREGROUND
root     24496  0.0  0.0 112652   964 pts/0    S+   16:31   0:00 grep --color=auto httpd

• My VM has 2GB memory and 2 Processors

Read details of this configure in the above-mentioned reference.

Restart Httpd

 # sudo systemctl restart httpd.service 

Inspect Status

 # systemctl status httpd.service ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2017-01-12 13:40:12 CST; 2h 48min ago Docs: man:httpd(8) man:apachectl(8) Main PID: 16576 (httpd) Status: "Total requests: 3625; Current requests/sec: -70.2; Current traffic: 0 B/sec" Memory: 869.7M CGroup: /system.slice/httpd.service ├─16576 /usr/sbin/httpd -DFOREGROUND ├─24122 /usr/sbin/httpd -DFOREGROUND ├─24125 /usr/sbin/httpd -DFOREGROUND ├─24154 /usr/sbin/httpd -DFOREGROUND ├─24157 /usr/sbin/httpd -DFOREGROUND ├─24211 /usr/sbin/httpd -DFOREGROUND ├─24219 /usr/sbin/httpd -DFOREGROUND └─24236 /usr/sbin/httpd -DFOREGROUND Jan 12 13:40:12 192-168-1-68.node systemd[1]: Starting The Apache HTTP Server... Jan 12 13:40:12 192-168-1-68.node httpd[16576]: [Thu Jan 12 13:40:12.614941 2017] [so:warn] [pid 16576] AH01574: module wsgi_module is already loaded, skipping Jan 12 13:40:12 192-168-1-68.node systemd[1]: Started The Apache HTTP Server. 

or

 # ps aux|grep httpd root 16576 0.0 0.0 232232 504 ? Ss 13:40 0:01 /usr/sbin/httpd -DFOREGROUND apache 24296 10.9 4.8 542936 91500 ? S 16:29 0:16 /usr/sbin/httpd -DFOREGROUND apache 24310 10.6 4.7 542804 89752 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND apache 24343 11.2 6.5 542804 123944 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND apache 24344 11.1 6.5 542804 124084 ? S 16:29 0:14 /usr/sbin/httpd -DFOREGROUND apache 24345 11.6 6.7 542804 126616 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND apache 24347 11.0 6.6 542804 125764 ? S 16:29 0:14 /usr/sbin/httpd -DFOREGROUND apache 24377 12.0 6.7 542804 126396 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND apache 24378 12.4 6.7 542804 126448 ? S 16:29 0:15 /usr/sbin/httpd -DFOREGROUND apache 24408 12.3 6.6 542804 126032 ? S 16:29 0:14 /usr/sbin/httpd -DFOREGROUND apache 24450 18.6 5.4 511560 102520 ? R 16:30 0:10 /usr/sbin/httpd -DFOREGROUND apache 24458 19.4 4.7 498476 89548 ? S 16:30 0:08 /usr/sbin/httpd -DFOREGROUND apache 24466 21.4 4.5 495424 86416 ? R 16:31 0:07 /usr/sbin/httpd -DFOREGROUND apache 24484 0.5 0.3 236456 5760 ? S 16:31 0:00 /usr/sbin/httpd -DFOREGROUND root 24496 0.0 0.0 112652 964 pts/0 S+ 16:31 0:00 grep --color=auto httpd 

My apache servers get a request at 2 per second . Before, the numbers of httpd processes increased until the system halted. Aftering setting the MPM Prefork, it works fine.