[英]MySQL I/O bound InnoDB query optimization problem without setting innodb_buffer_pool_size to 5GB
I got myself into a MySQL design scalability issue. 我陷入了MySQL设计可扩展性问题。 Any help would be greatly appreciated.
任何帮助将不胜感激。
The requirements: 要求:
Storing users' SOCIAL_GRAPH and USER_INFO about each user in their social graph. 存储用户在其社交图中关于每个用户的SOCIAL_GRAPH和USER_INFO。 Many concurrent reads and writes per second occur.
每秒发生许多并发读取和写入。 Dirty reads acceptable.
脏读是可以接受的。
Current design: 当前设计:
We have 2 (relevant) tables. 我们有2个(相关)表。 Both InnoDB for row locking, instead of table locking.
两种InnoDB都用于行锁定,而不是表锁定。
USER_SOCIAL_GRAPH table that maps a logged in (user_id) to another (related_user_id). USER_SOCIAL_GRAPH表将已登录的(user_id)映射到另一个(related_user_id)。 PRIMARY key composite user_id and related_user_id.
主键组合user_id和related_user_id。
USER_INFO table with information about each related user. USER_INFO表,其中包含有关每个相关用户的信息。 PRIMARY key is (related_user_id).
主键为(related_user_id)。
Note 1: No relationships defined. 注1:未定义关系。
Note 2: Each table is now about 1GB in size, with 8 million and 2 million records, respectively. 注2:现在每个表的大小约为1GB,分别具有800万条记录和200万条记录。
Simplified table SQL creates: 简化的表SQL创建:
CREATE TABLE `user_social_graph` (
`user_id` int(10) unsigned NOT NULL,
`related_user_id` int(11) NOT NULL,
PRIMARY KEY (`user_id`,`related_user_id`),
KEY `user_idx` (`user_id`)
) ENGINE=InnoDB;
CREATE TABLE `user_info` (
`related_user_id` int(10) unsigned NOT NULL,
`screen_name` varchar(20) CHARACTER SET latin1 DEFAULT NULL,
[... and many other non-indexed fields irrelevant]
`last_updated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`related_user_id`),
KEY `last_updated_idx` (`last_updated`)
) ENGINE=InnoDB;
MY.CFG values set: MY.CFG值设置为:
innodb_buffer_pool_size = 256M
key_buffer_size = 320M
Note 3: Memory available 1GB, these 2 tables are 2GBs, other innoDB tables 3GB. 注3:可用内存为1GB,这两个表为2GB,其他innoDB表为3GB。
Problem: 问题:
The following example SQL statement, which needs to access all records found, takes 15 seconds to execute (!!) and num_results = 220,000: 以下示例SQL语句需要访问找到的所有记录,需要15秒执行(!!),并且num_results = 220,000:
SELECT SQL_NO_CACHE COUNT(u.related_user_id)
FROM user_info u LEFT JOIN user_socialgraph u2 ON u.related_user_id = u2.related_user_id
WHERE u2.user_id = '1'
AND u.related_user_id = u2.related_user_id
AND (NOT (u.related_user_id IS NULL));
For a user_id with a count of 30,000, it takes about 3 seconds (!). 对于计数为30,000的user_id,大约需要3秒(!)。
EXPLAIN EXTENDED for the 220,000 count user. 对220,000个计数用户的EXPLAIN EXTENDED。 It uses indices:
它使用索引:
+----+-------------+-------+--------+------------------------+----------+---------+--------------------+--------+----------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+--------+------------------------+----------+---------+--------------------+--------+----------+--------------------------+
| 1 | SIMPLE | u2 | ref | user_user_idx,user_idx | user_idx | 4 | const | 157320 | 100.00 | Using where |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | u2.related_user_id | 1 | 100.00 | Using where; Using index |
+----+-------------+-------+--------+------------------------+----------+---------+--------------------+--------+----------+--------------------------+
How do we speed these up without setting innodb_buffer_pool_size to 5GB? 如何在不将innodb_buffer_pool_size设置为5GB的情况下加快速度?
Thank you! 谢谢!
The user_social_graph table is not indexed correctly !!! user_social_graph表未正确索引!!!
You have ths: 您有:
CREATE TABLE user_social_graph
创建表
user_social_graph
( user_id
int(10) unsigned NOT NULL, (
user_id
int(10)unsigned NOT NULL,
related_user_id
int(11) NOT NULL, related_user_id
int(11)NOT NULL,
PRIMARY KEY ( user_id
, related_user_id
), 主键(
user_id
, related_user_id
),
KEY user_idx
( user_id
)) 密钥
user_idx
( user_id
))
ENGINE=InnoDB; ENGINE = InnoDB的;
The second index is redundant since the first column is user_id. 由于第一列是user_id,因此第二个索引是冗余的。 You are attempting to join the related_user_id column over to the user_info table.
您正在尝试将related_user_id列加入到user_info表中。 That column needed to be indexed.
该列需要索引。
Change user_social_graphs as follows: 如下更改user_social_graphs:
CREATE TABLE user_social_graph
创建表
user_social_graph
( user_id
int(10) unsigned NOT NULL, (
user_id
int(10)unsigned NOT NULL,
related_user_id
int(11) NOT NULL, related_user_id
int(11)NOT NULL,
PRIMARY KEY ( user_id
, related_user_id
), 主键(
user_id
, related_user_id
),
UNIQUE KEY related_user_idx
( related_user_id
, user_id
)) 唯一键
related_user_idx
( related_user_id
, user_id
))
ENGINE=InnoDB; ENGINE = InnoDB的;
This should change the EXPLAIN PLAN. 这应该更改“解释计划”。 Keep in mind that the index order matters depending the the way you query the columns.
请记住,索引顺序很重要,具体取决于查询列的方式。
Give it a Try !!! 试试看 !!!
What is the MySQL version? 什么是MySQL版本? Its manual contains important information for speeding up statements and code in general;
它的手册包含重要的信息,这些信息通常可以加快语句和代码的速度。
Change your paradigm to a data warehouse capable to manage till terabyte table. 将您的范例更改为能够管理到TB级表的数据仓库。 Migrate your legacy MySQL data base with free tool or application to the new paradigm.
使用免费工具或应用程序将旧版MySQL数据库迁移到新范例中。 This is an example: http://www.infobright.org/Downloads/What-is-ICE/ many others (free and commercial).
这是一个示例: http : //www.infobright.org/Downloads/What-is-ICE/许多其他(免费和商业)。
PostgreSQL is not commercial and there a lot of tools to migrate MySQL to it! PostgreSQL不是商业性的,有很多工具可以将MySQL迁移到它!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.