MySQL 处理大量数据
[英]MySQL to handle large amount of data
问题描述
Currently my application is generating around 40million records every hour, i have created a partition for every hour so it will be easier for me to drop the partition as and when required, also using the partition to aggregate the data.
目前我的应用程序每小时生成大约 4000 万条记录，我每小时创建一个分区，因此我可以在需要时更轻松地删除分区，也可以使用分区来聚合数据。
My aggregation procedure will run fine, if there are not any queries happening but once those queries are started, the aggregation code takes more than an hour to complete.我的聚合过程将运行良好，如果没有任何查询发生，但一旦这些查询开始，聚合代码需要一个多小时才能完成。
in MySQL is there a process to freeze and not impact based on the queries happening to database?在 MySQL 中是否有一个进程冻结而不影响数据库发生的查询？
1 个解决方案
解决方案1
0 2021-11-23 21:53:53
Let's see the aggregation code.让我们看看聚合代码。 And SHOW CREATE TABLE .并SHOW CREATE TABLE 。
Other things to try:其他要尝试的事情：
- Aggregate every 5 minutes instead.而是每 5 分钟聚合一次。
- Aggregate "continually". “不断地”聚合。 That is, do all the rows since you last did an aggregation.也就是说，执行自上次聚合以来的所有行。
- Collect the rows into a 'staging' table.将行收集到“暂存”表中。 Then do the aggregation from that and finally copy the rows into the main table.然后从中进行聚合，最后将行复制到主表中。 Here's a discussion of a "ping-pong" technique for such: http://mysql.rjweb.org/doc.php/staging_table以下是对此类“乒乓”技术的讨论： http://mysql.rjweb.org/doc.php/staging_table
11K rows per second is a lot, but not impossible.每秒 11K 行是很多，但并非不可能。
Further questions:进一步的问题：
- How much RAM do you have?你有多少内存？
- What is the setting of
innodb_buffer_pool_size?innodb_buffer_pool_size的设置是什么？ - Do you have SSD?你有SSD吗？
- What type of reads are going on at the same time?什么类型的读取同时进行？ (reading from big table? reading from summary table?) （从大表中读取？从汇总表中读取？）
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.