Joomla database memory leak

Question

My client has got a pretty large Joomla-based website hosted on Amazon EC2 with 1.5GB of RAM. The server hosts both Apache and MySQL. Right now the database size is around 250MB and the website gets daily traffic of about 5000. It looks like there is a severe memory leak on the website as sometimes MySQL uses about 99% of CPU memory and then crashes. I have tried optimizing database tables, modifying my .cnf , but still there is no improvement.

There are finder tables used by Joomla smart search which occupy over 100MB of db size. I have disabled smart search, but still the problem occurs.

Friends, please throw some suggestions in fixing this.

Thanks.

Below is the my.cnf file

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

symbolic-links=0
bind-address            = 127.0.0.1

default-storage-engine=innodb
transaction-isolation    =   REPEATABLE-READ
character-set-server    =   UTF8
collation-server      =   UTF8_general_ci

max_connections         = 5000
wait_timeout            = 30
connect_timeout         = 60
#interactive_timeout     = 600
#max_connect_errors      = 1000000
#max_allowed_packet      = 10M


skip-external-locking
key_buffer_size = 384M
max_allowed_packet = 1M
table_open_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8

slow_query_log
long_query_time = 2

[mysqld_safe]
log-error=/var/log/mysqld.log
myisam_sort_buffer_size = 64M

Answer 1

My bet would be that you are being hit by a rogue robot - one of the many SEO spiders out there, or tools like 80legs that let people program a network of bots to carry out tasks - often with errors in their programming that result in a heavy bombardment.

I can never remember which of the MySQL settings take memory once and which are per connection - but as you are set to allow up to 5000 simultaneous connections and some of the buffers are 2 and 8 MB I'd bet that the total memory usage under heavy load could easily be in excess of the total ram available.

Your current settings would allow all of your daily traffic to hit simultaneously. I'd knock that down to a setting of a hundred or less and see if that gives more stability.

There are various MySQL tuner scripts out there that can help you spot where too much memory is allocated.

If you have access logs from around the time of the crashes / high load I'd check for malicious bots though - we've had a constant battle to reign them in on some sites we monitor/control.

You might also check the thread_concurrency value - depending upon how many CPUs you have available.