What Could be Causing CPU Spikes every 5 minutes with MySQL on RDS?

Question

Under a relatively write intensive constant load we are experiencing CPU spikes every 5 minutes using MySQL on RDS. Has anyone experienced something similar or have any idea what might be causing it?

MySQL Version: 5.6.22 Storage Engine: InnoDB

We experience the spikes on both M3 Medium and M3 Large instance types, with both magnetic and SSD disks, and they occur whether or not we have multi AZ enabled.

The more writes we have, the steeper the spikes are. However, the number of writes does not seem to impact the frequency. They continue a little while after we finish writing. They do not appear under a read only load.

Here are the InnoDB variables:

innodb_adaptive_flushing: ON
innodb_adaptive_flushing_lwm: 10
innodb_adaptive_hash_index: ON
innodb_adaptive_max_sleep_delay: 150000
innodb_additional_mem_pool_size: 8388608
innodb_api_bk_commit_interval: 5
innodb_api_disable_rowlock: OFF
innodb_api_enable_binlog: OFF
innodb_api_enable_mdl: OFF
innodb_api_trx_level: 0
innodb_autoextend_increment: 64
innodb_autoinc_lock_mode: 1
innodb_buffer_pool_dump_at_shutdown: OFF
innodb_buffer_pool_dump_now: OFF
innodb_buffer_pool_filename: ib_buffer_pool
innodb_buffer_pool_instances: 8
innodb_buffer_pool_load_abort: OFF
innodb_buffer_pool_load_at_startup: OFF
innodb_buffer_pool_load_now: OFF
innodb_buffer_pool_size: 5705302016
innodb_change_buffer_max_size: 25
innodb_change_buffering: all
innodb_checksum_algorithm: innodb
innodb_checksums: ON
innodb_cmp_per_index_enabled: OFF
innodb_commit_concurrency: 0
innodb_compression_failure_threshold_pct: 5
innodb_compression_level: 6
innodb_compression_pad_pct_max: 50
innodb_concurrency_tickets: 5000
innodb_data_file_path: ibdata1:12M:autoextend
innodb_data_home_dir: /rdsdbdata/db/innodb
innodb_disable_sort_file_cache: OFF
innodb_doublewrite: ON
innodb_fast_shutdown: 1
innodb_file_format: Antelope
innodb_file_format_check: ON
innodb_file_format_max: Antelope
innodb_file_per_table: ON
innodb_flush_log_at_timeout: 1
innodb_flush_log_at_trx_commit: 1
innodb_flush_method: O_DIRECT
innodb_flush_neighbors: 1
innodb_flushing_avg_loops: 30
innodb_force_load_corrupted: OFF
innodb_force_recovery: 0
innodb_ft_aux_table: 
innodb_ft_cache_size: 8000000
innodb_ft_enable_diag_print: OFF
innodb_ft_enable_stopword: ON
innodb_ft_max_token_size: 84
innodb_ft_min_token_size: 3
innodb_ft_num_word_optimize: 2000
innodb_ft_result_cache_limit: 2000000000
innodb_ft_server_stopword_table: 
innodb_ft_sort_pll_degree: 2
innodb_ft_total_cache_size: 640000000
innodb_ft_user_stopword_table: 
innodb_io_capacity: 200
innodb_io_capacity_max: 2000
innodb_large_prefix: OFF
innodb_lock_wait_timeout: 50
innodb_locks_unsafe_for_binlog: OFF
innodb_log_buffer_size: 8388608
innodb_log_compressed_pages: ON
innodb_log_file_size: 134217728
innodb_log_files_in_group: 2
innodb_log_group_home_dir: /rdsdbdata/log/innodb
innodb_lru_scan_depth: 1024
innodb_max_dirty_pages_pct: 75
innodb_max_dirty_pages_pct_lwm: 0
innodb_max_purge_lag: 0
innodb_max_purge_lag_delay: 0
innodb_mirrored_log_groups: 1
innodb_monitor_disable: 
innodb_monitor_enable: 
innodb_monitor_reset: 
innodb_monitor_reset_all: 
innodb_old_blocks_pct: 37
innodb_old_blocks_time: 1000
innodb_online_alter_log_max_size: 134217728
innodb_open_files: 2000
innodb_optimize_fulltext_only: OFF
innodb_page_size: 16384
innodb_print_all_deadlocks: OFF
innodb_purge_batch_size: 300
innodb_purge_threads: 1
innodb_random_read_ahead: OFF
innodb_read_ahead_threshold: 56
innodb_read_io_threads: 4
innodb_read_only: OFF
innodb_replication_delay: 0
innodb_rollback_on_timeout: OFF
innodb_rollback_segments: 128
innodb_sort_buffer_size: 1048576
innodb_spin_wait_delay: 6
innodb_stats_auto_recalc: ON
innodb_stats_method: nulls_equal
innodb_stats_on_metadata: OFF
innodb_stats_persistent: ON
innodb_stats_persistent_sample_pages: 20
innodb_stats_sample_pages: 8
innodb_stats_transient_sample_pages: 8
innodb_status_output: OFF
innodb_status_output_locks: OFF
innodb_strict_mode: OFF
innodb_support_xa: ON
innodb_sync_array_size: 1
innodb_sync_spin_loops: 30
innodb_table_locks: ON
innodb_thread_concurrency: 0
innodb_thread_sleep_delay: 10000
innodb_undo_directory: .
innodb_undo_logs: 128
innodb_undo_tablespaces: 0
innodb_use_native_aio: ON
innodb_use_sys_malloc: ON
innodb_version: 5.6.22
innodb_write_io_threads: 4

Thanks!

Answer 1

I assume the VARIABLES you provided are for m3.large ?

For SSDs, I'm pretty sure innodb_flush_neighbors should be 0. (I'm surprised Amazon did not catch that.) http://dev.mysql.com/doc/refman/5.6/en/optimizing-innodb-diskio.html

Try increasing innodb_flushing_avg_loops to 200. "Increasing the value makes the rate of flush operations change smoothly and gradually as the workload changes."

innodb_read_io_threads and innodb_write_io_threads might be better at 8.

The "adaptive_flushing" sounds relevant, but I don't know enough about it to comment.

If those are not sufficient, please

• explain the processing some more,
• state whether there are any cron jobs running */5 ,
• and any other clues.

Also, you could provide these for a more thorough study of the settings. (I did not ask for them because I do not have any good clues of where to look for the spike.)

• SHOW VARIABLES;
• SHOW GLOBAL STATUS; -- assuming it has been up at least a day.