MariaDb10（MySQL 5.6）-從MySQL 5.5更新后，隨機的高I / O和慢查詢
[英]MariaDb10 (MySQL 5.6) - random high I/O and slow queries after update from MySQL 5.5
問題描述
我們使用MySQL數據庫運行Java Web應用程序，並使用帶有c3p0連接池的5.1 MySQL連接器進行連接。 白天收集的數據將在一夜之間轉換為數據倉庫。 應用程序和數據庫都生活在具有4GB RAM（使用gp2（SSD）EBS卷）的同一EC2實例（m3.medium）上。
將數據庫從MySQL 5.5升級到MariaDB 10.0.17（也嘗試過MySQL 5.6.23）會導致在某些情況下每晚進行數據倉庫構建花費的時間過長。
數據倉庫構建基於運行SQL查詢和JRuby腳本的Maven / Ant腳本集。 數據倉庫（DWH）構建由CRON作業執行一整夜，並且在運行DWH構建時不會停止應用程序。 DWH構建首先刪除整個dwh模式，然后重建它。
在這些條件下，針對MySQL 5.5運行DWH構建大約需要4個小時，而對於相同的數據，針對MariaDB 10（和Mysql 5.6）有時則需要多達16個小時。 但是，這種行為並不一致，有時使用MariaDB 10會在4小時內構建DWH。
這些測試是在Java應用程序僅連接到數據庫而不對數據庫進行任何查詢的測試環境中運行的，因此它不會以任何方式增加數據庫負載。
特別是，DWH構建過程中的兩個查詢有時需要花費更長的時間才能運行（總計32K秒對4K秒）。
當DWH在這些查詢上“卡住”時，針對ibdata1 I / O增加，並且MariaDB使用90％以上的I / O（基於iotop）。
我花了很多時間測試不同的MariaDB配置（更改分配給MariaDB的內存，線程數等），但結果始終是相同的。
從MySQL 5.5升級到5.6時，還有其他人遇到過類似的問題嗎？ 是否有其他建議可以嘗試找出可能是什么問題的想法？
在這里您還可以找到磁盤I / O使用率（Ops / s）的圖表。 在3/24和3/26，DWH正確運行（4小時），在3/25，DWH花費了四倍（16小時）。
讀取： 讀取操作/秒
寫入： 寫入操作/秒
如您所見，即使數據和進程相同，I / O配置文件也完全不同，就好像MariaDB在做一些額外的工作一樣。
謝謝！
下面是我在其中一個查詢運行緩慢時收集的一些信息。
顯示完整的進程列表（查詢57是DWH構建過程，其他所有內容都屬於Java應用程序）：
| Id | User | Host | db | Command | Time | State | Info | Progress |
| 50 | steve | localhost:37029 | qrtz | Sleep | 3 | | NULL | 0.000 |
| 51 | steve | localhost:37030 | qrtz | Sleep | 3 | | NULL | 0.000 |
| 52 | steve | localhost:37031 | qrtz | Sleep | 3 | | NULL | 0.000 |
| 57 | steve | localhost:37366 | dwh | Query | 14222 | Sending data | INSERT INTO dwh.assessment_answer_fact ( page_id, page_number, category_id, category_number, category_title, question_id, question_display_number, question_number, question_text, question_identifier, answer_value, assessment_fact_id ) SELECT p.id, p.page_number, ic.id, ic.category_number, ic.title, q.id, q.question_display_number, q.question_number, q.question_text, q.question_identifier, av.value, ca.id FROM smr.cans_assessment ca join survey.surveys s ON ( s.id = ca.survey_id ) join survey.instruments i ON ( s.instrument_id = i.id ) join survey.survey_categories sc ON ( sc.survey_id = s.id ) join survey.instrument_categories ic ON ( sc.instrument_category_id = ic.id ) join survey.pages p ON ( p.id = ic.page_id ) join survey.answers a ON ( a.survey_category_id = sc.id ) join survey.answer_values av ON ( av.answer_id = a.id ) join survey.questions q ON ( a.question_id = q.id ) WHERE ca.id in (select af.assessment_id from dwh.assessment_fact af) | 0.000 |
| 64 | steve | localhost:37700 | NULL | Sleep | 5394 | | NULL | 0.000 |
| 65 | steve | localhost:37701 | NULL | Sleep | 5394 | | NULL | 0.000 |
| 66 | steve | localhost:37702 | NULL | Sleep | 5394 | | NULL | 0.000 |
| 67 | steve | localhost:37703 | NULL | Sleep | 5394 | | NULL | 0.000 |
| 68 | steve | localhost:37770 | NULL | Query | 0 | init | show full processlist | 0.000 |
顯示引擎的innodb狀態 ：
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 28198 srv_active, 0 srv_shutdown, 35270 srv_idle
srv_master_thread log flush and writes: 63465
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 247509
OS WAIT ARRAY INFO: signal count 230685
Mutex spin waits 233790, rounds 7015555, OS waits 233481
RW-shared spins 5789, rounds 173633, OS waits 5403
RW-excl spins 3842, rounds 258871, OS waits 7270
Spin rounds per wait: 30.01 mutex, 29.99 RW-shared, 67.38 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 2795813
Purge done for trx's n:o < 2795733 undo n:o < 0 state: running but idle
History list length 327
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 68, OS thread handle 0x7fa659961700, query id 135532 localhost 127.0.0.1 steve init
show engine innodb status
---TRANSACTION 2773754, not started
MySQL thread id 52, OS thread handle 0x7fa6923aa700, query id 135518 localhost 127.0.0.1 steve cleaning up
---TRANSACTION 2795812, not started
MySQL thread id 51, OS thread handle 0x7fa6923f3700, query id 135531 localhost 127.0.0.1 steve cleaning up
---TRANSACTION 2773251, not started
MySQL thread id 1, OS thread handle 0x7fa6924ce700, query id 0 Waiting for background binlog tasks
---TRANSACTION 2794864, ACTIVE 14118 sec fetching rows
mysql tables in use 11, locked 11
136069 lock struct(s), heap size 20477480, 23704803 row lock(s), undo log entries 3791652
MySQL thread id 57, OS thread handle 0x7fa69243c700, query id 129742 localhost 127.0.0.1 steve Sending data
INSERT INTO dwh.assessment_answer_fact ( page_id, page_number, category_id, category_number, category_title, question_id, question_display_number, question_number, question_text, question_identifier, answer_value, assessment_fact_id ) SELECT p.id, p.page_number, ic.id, ic.category_number, ic.title, q.id, q.question_display_number, q.question_number, q.question_text, q.question_identifier, av.value, ca.id FROM smr.cans_assessment ca join survey.surveys s ON ( s.id = ca.survey_id ) join survey.instruments i ON ( s.instrument_id = i.id ) join survey.survey_categories sc ON ( sc.survey_id = s.id
Trx #rec lock waits 0 #table lock waits 0
Trx total rec lock wait time 0 SEC
Trx total table lock wait time 0 SEC
TABLE LOCK table `survey`.`questions` trx id 2794864 lock mode IS lock hold time 14118 wait time before grant 0
RECORD LOCKS space id 852 page no 9 n bits 184 index `PRIMARY` of table `survey`.`questions` trx table locks 12 total table locks 1 trx id 2794864 lock mode S lock hold time 14118 wait time before grant 0
TABLE LOCK table `survey`.`answers` trx id 2794864 lock mode IS lock hold time 14118 wait time before grant 0
RECORD LOCKS space id 861 page no 12 n bits 824 index `FKCD7DB87560DC6F04` of table `survey`.`answers` trx table locks 12 total table locks 1 trx id 2794864 lock mode S lock hold time 14118 wait time before grant 0
RECORD LOCKS space id 861 page no 6 n bits 336 index `PRIMARY` of table `survey`.`answers` trx table locks 12 total table locks 1 trx id 2794864 lock mode S locks rec but not gap lock hold time 14118 wait time before grant 0
TABLE LOCK table `survey`.`survey_categories` trx id 2794864 lock mode IS lock hold time 14118 wait time before grant 0
RECORD LOCKS space id 855 page no 7 n bits 328 index `PRIMARY` of table `survey`.`survey_categories` trx table locks 12 total table locks 1 trx id 2794864 lock mode S locks rec but not gap lock hold time 14118 wait time before grant 0
TABLE LOCK table `smr`.`cans_assessment` trx id 2794864 lock mode IS lock hold time 14118 wait time before grant 0
RECORD LOCKS space id 1800 page no 42 n bits 824 index `FK52F1DDEAF118504` of table `smr`.`cans_assessment` trx table locks 12 total table locks 1 trx id 2794864 lock mode S locks gap before rec lock hold time 14118 wait time before grant 0
RECORD LOCKS space id 861 page no 32512 n bits 336 index `PRIMARY` of table `survey`.`answers` trx table locks 12 total table locks 1 trx id 2794864 lock mode S locks rec but not gap lock hold time 14118 wait time before grant 0
TOO MANY LOCKS PRINTED FOR THIS TRX: SUPPRESSING FURTHER PRINTS
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
29827256 OS file reads, 1144210 OS file writes, 345551 OS fsyncs
631.07 reads/s, 16384 avg bytes/read, 20.61 writes/s, 9.37 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 858, free list len 5697, seg size 6556, 299256 merges
merged operations:
insert 7839828, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
2106.29 hash searches/s, 2190.81 non-hash searches/s
---
LOG
---
Log sequence number 184023146842
Log flushed up to 184023103841
Pages flushed up to 184006513482
Last checkpoint at 184006296459
Max checkpoint age 216721613
Checkpoint age target 209949063
Modified age 16633360
Checkpoint age 16850383
0 pending log writes, 0 pending chkp writes
42065 log i/o's done, 1.22 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 549453824; in additional pool allocated 0
Total memory allocated by read views 512
Internal hash tables (constant factor + variable factor)
Adaptive hash index 10673872 (8851048 + 1822824)
Page hash 553976 (buffer pool 0 only)
Dictionary cache 4009988 (2214224 + 1795764)
File system 914416 (812272 + 102144)
Lock system 21808096 (1329176 + 20478920)
Recovery system 0 (0 + 0)
Dictionary memory allocated 1795764
Buffer pool size 32767
Buffer pool size, bytes 536854528
Free buffers 965
Database pages 30442
Old database pages 11252
Modified db pages 1638
Percent of dirty pages(LRU & free pages): 5.215
Max dirty pages percent: 75.000
Pending reads 1
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 687161, not young 141371905
4.33 youngs/s, 3024.61 non-youngs/s
Pages read 29827185, created 180475, written 972592
631.09 reads/s, 4.10 creates/s, 15.67 writes/s
Buffer pool hit rate 952 / 1000, young-making rate 0 / 1000 not 232 / 1000
Pages read ahead 485.66/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 30442, unzip_LRU len: 0
I/O sum[8334]:cur[77], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
1 RW transactions active inside InnoDB
0 RO transactions active inside InnoDB
1 out of 1000 descriptors used
Main thread process no. 7359, id 140352388314880, state: sleeping
Number of rows inserted 7172945, updated 106861, deleted 0, read 1363672517
240.68 inserts/s, 0.00 updates/s, 0.00 deletes/s, 1666.92 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
從SELECT * FROM information_schema.metadata_lock_info （METADATA_LOCK_INFO插件）獲得的元數據鎖：
+-----------+-------------------------+-----------------+---------------------+--------------+------------------------+
| THREAD_ID | LOCK_MODE | LOCK_DURATION | LOCK_TYPE | TABLE_SCHEMA | TABLE_NAME |
+-----------+-------------------------+-----------------+---------------------+--------------+------------------------+
| 57 | MDL_INTENTION_EXCLUSIVE | MDL_STATEMENT | Global read lock | | |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | instrument_categories |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | questions |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | pages |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | answer_values |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | surveys |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | dwh | assessment_fact |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | answers |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | smr | cans_assessment |
| 57 | MDL_SHARED_WRITE | MDL_TRANSACTION | Table metadata lock | dwh | assessment_answer_fact |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | survey_categories |
| 57 | MDL_SHARED_READ | MDL_TRANSACTION | Table metadata lock | survey | instruments |
+-----------+-------------------------+-----------------+---------------------+--------------+------------------------+
pt-ioprofile -c大小的文件I / O（Percona工具包）：
total pread pwrite fsync filename
72269824 72269824 0 0 /data/mysql/smr/survey/answer_values.ibd
8323072 393216 7929856 0 /data/mysql/smr/ibdata1
4964352 770048 4194304 0 /data/mysql/smr/dwh/assessment_answer_fact.ibd
4833280 4833280 0 0 /data/mysql/smr/survey/answers.ibd
2707456 0 2707456 0 /data/mysql/smr/ib_logfile1
1572864 1572864 0 0 /data/mysql/smr/dwh/assessment_fact.ibd
1523712 1523712 0 0 /data/mysql/smr/survey/surveys.ibd
1114112 1114112 0 0 /data/mysql/smr/survey/survey_categories.ibd
163840 163840 0 0 /data/mysql/smr/smr/cans_assessment.ibd
49152 49152 0 0 /data/mysql/smr/survey/questions.ibd
0 0 0 0 /data/mysql/smr/ib_logfile0
一些MariaDB配置參數 （顯示全局變量太長而無法容納在消息中）：
log_bin
binlog_format = mixed
max_binlog_size = 1073741824
binlog_cache_size = 4M
innodb_buffer_pool_size = 512M
innodb_io_capacity = 100
query_cache_size = 64M
innodb_flush_neighbors = 0
innodb_flush_log_at_trx_commit = 0
tmp_table_size = 32M
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
max_allowed_packet = 16M
innodb_file_per_table = 1
查詢速度慢 ：
INSERT INTO dwh.assessment_answer_fact ( page_id, page_number, category_id, category_number, category_title, question_id, question_display_number, question_number, question_text, question_identifier, answer_value, assessment_fact_id ) SELECT p.id, p.page_number, ic.id, ic.category_number, ic.title, q.id, q.question_display_number, q.question_number, q.question_text, q.question_identifier, av.value, ca.id FROM smr.cans_assessment ca join survey.surveys s ON ( s.id = ca.survey_id ) join survey.instruments i ON ( s.instrument_id = i.id ) join survey.survey_categories sc ON ( sc.survey_id = s.id ) join survey.instrument_categories ic ON ( sc.instrument_category_id = ic.id ) join survey.pages p ON ( p.id = ic.page_id ) join survey.answers a ON ( a.survey_category_id = sc.id ) join survey.answer_values av ON ( av.answer_id = a.id ) join survey.questions q ON ( a.question_id = q.id ) WHERE ca.id in (select af.assessment_id from dwh.assessment_fact af)
解釋慢查詢 ：
+------+-------------+-------+--------+-----------------------------------------------+--------------------+---------+----------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+--------+-----------------------------------------------+--------------------+---------+----------------------------------+------+-------------+
| 1 | PRIMARY | q | ALL | PRIMARY | NULL | NULL | NULL | 384 | |
| 1 | PRIMARY | a | ref | PRIMARY,FKCD7DB8757DA6A899,FKCD7DB87560DC6F04 | FKCD7DB87560DC6F04 | 8 | survey.q.id | 321 | |
| 1 | PRIMARY | sc | eq_ref | PRIMARY,FKBEB7B82168B10433,FKBEB7B821AF118504 | PRIMARY | 8 | survey.a.survey_category_id | 1 | |
| 1 | PRIMARY | ca | ref | PRIMARY,FK52F1DDEAF118504 | FK52F1DDEAF118504 | 8 | survey.sc.survey_id | 1 | Using index |
| 1 | PRIMARY | af | eq_ref | PRIMARY | PRIMARY | 8 | smr.ca.id | 1 | Using index |
| 1 | PRIMARY | s | eq_ref | PRIMARY,FK91914459785448E4 | PRIMARY | 8 | survey.sc.survey_id | 1 | |
| 1 | PRIMARY | i | eq_ref | PRIMARY | PRIMARY | 8 | survey.s.instrument_id | 1 | Using index |
| 1 | PRIMARY | ic | eq_ref | PRIMARY,FK1118917415F49764 | PRIMARY | 8 | survey.sc.instrument_category_id | 1 | |
| 1 | PRIMARY | av | ref | FK5075C8C38297FA84 | FK5075C8C38297FA84 | 8 | survey.a.id | 1 | |
| 1 | PRIMARY | p | eq_ref | PRIMARY | PRIMARY | 8 | survey.ic.page_id | 1 | |
+------+-------------+-------+--------+-----------------------------------------------+--------------------+---------+----------------------------------+------+-------------+
行數 ：
| answer_values | 10766276 |
| answers | 12020566 |
| answers_answer_values | 11662057 |
| cans_assessment | 77221 |
| instrument_categories | 85 |
| instruments | 11 |
| pages | 11 |
| questions | 384 |
| survey_categories | 1462377 |
| survey_categories_answers | 10954877 |
| surveys | 118702 |
| surveys_survey_categories | 1515111 |
| assessment_fact | 76803 |
| assessment_answer_fact | 7673695 |
2 個解決方案
解決方案1
0 2015-03-26 06:09:09
真正的問題是SELECT ：
SELECT p.id, p.page_number, ic.id, ic.category_number, ic.title,
q.id, q.question_display_number, q.question_number, q.question_text,
q.question_identifier, av.value, ca.id
FROM smr.cans_assessment ca
join survey.surveys s ON ( s.id = ca.survey_id )
join survey.instruments i ON ( s.instrument_id = i.id )
join survey.survey_categories sc ON ( sc.survey_id = s.id )
join survey.instrument_categories ic ON ( sc.instrument_category_id = ic.id )
join survey.pages p ON ( p.id = ic.page_id )
join survey.answers a ON ( a.survey_category_id = sc.id )
join survey.answer_values av ON ( av.answer_id = a.id )
join survey.questions q ON ( a.question_id = q.id )
WHERE ca.id in (
SELECT af.assessment_id
from dwh.assessment_fact af
...
請提供整個SELECT（在輸出中被截斷）。
請檢查JOIN是否正在通過索引。
請提供EXPLAIN SELECT ...
如有可能，將in ( SELECT ... )輸入JOIN。 （這可能是主要的性能問題！）
什么版本的？ 不僅是“ 10”和“ 5.6”。 （需要查看它是否具有in ( SELECT ... )優化。舊版本對此進行了極大的優化。）
多少內存？ 除非您有一台小型計算機，否則0.5GB的buffer_pool會很低。
請解釋一下INSERT..SELECT的意圖。
innodb_file_per_table的設置是什么？ （這與為什么ibdata1上可能有很多I / O的原因有關。）
這些表有多大，包括您要插入其中的表？
編輯1
假設這是整個尾隨子句：
WHERE ca.id in (
SELECT af.assessment_id
from dwh.assessment_fact af )
您是否看到刪除WHERE子句並添加它會加快查詢速度：
JOIN dwh.assessment_fact af ON af.assessment_id = ca.id
解決方案2
0 2015-03-27 16:22:04
這個“答案”集中在
DWH構建首先刪除整個dwh模式，然后重建它。
我建議那是倒退。 如果執行以下操作，則沒有停機時間，因此該緩慢的查詢沒有明顯的延遲：
-
CREATE TABLE new_1LIKEreal_1; -每張要重建的桌子一張 - 將數據加載到新表中。
- 在
new表中執行討厭的INSERT..SELECT。 -
RENAME TABLE real_1 TO old_1, new_1 TO real_1, real_2 TO old_2, ...;-這是唯一的“停機時間”，並且是瞬時的。 -
DROP TABLE old_1;-對於每個舊桌子。
在 AWS 上從 MySQL 5.5 升級到 5.6 時 Hibernate 和日期的問題
[英]Problems with Hibernate and dates when upgrading from MySQL 5.5 to 5.6 on AWS
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.