堆棧內存溢出
  簡體   English   中英

MySQL 連接和行太多 - 執行時間

[英]MySQL too many joins and rows - execution time

 原文  2020-04-04 03:54:12  0   3    php/ mysql

問題描述

我正在嘗試運行查詢，但連接太多，因此執行時間很慢。 我正在尋找其他替代方案，例如讓 PHP 處理它而不是 mysql。 仍然在為我必須運行多少個查詢/連接而絞盡腦汁。 有什么建議么？

5 表及其行
2Variations 每年增加 5000 行
di 每年可能增加 100,000
文檔每年增加 50,000
type 和 hashtag 分別是大約 20 個常量

變量
現在這里有我的查詢
$groups = 按類型分組或按主題標簽分組或兩者
$query_where = REGEXP 僅從數組列表（由用戶選擇）中查找特定類型和/或主題標簽，類似於 find_in_set

$query_where .=  "AND CONCAT(',', `type.product_type`, ',') REGEXP ',($types_arr),' ";

下面的查詢運行大約38 秒

        SELECT 
type.product_type as type, hashtag.variation_hashtag as hashtag,
di.warehouse, sum(di.qty)as qty, sum(di.price) as price, sum(di.sold_price) as sold_price, sum(di.cogs) as cogs, avg(di.price) as ave_price, avg(di.sold_price) as ave_sold_price, 
YEAR(docs.created_on) as year, MONTH(docs.created_on) as month
        FROM  2Variations v
            LEFT JOIN docs_inventory as di on di.product_id = v.product_id
            LEFT JOIN docs on docs.id = di.doc_id
            LEFT JOIN variations_type_link as type on v.id = type.id
            LEFT JOIN variations_hashtag_link as hashtag on v.id = hashtag.id
        WHERE
            di.deleted = 0 and
            type.deleted = 0 and
            (hashtag.deleted = 0 or hashtag.deleted is null) and 
            di.warehouse between 1 and 2
            ".$query_where."
        GROUP BY
            YEAR(docs.created_on), MONTH(docs.created_on), ".$groups;

解釋

+----+-------------+---------+--------+--------------------+------------+---------+-----------------------+------+----------------------------------------------+--+
| id | select_type |  table  |  type  |   possible_keys    |    key     | key_len |          ref          | rows |                    Extra                     |  |
+----+-------------+---------+--------+--------------------+------------+---------+-----------------------+------+----------------------------------------------+--+
|  1 | SIMPLE      | type    | ALL    | PRIMARY,deleted    | NULL       | NULL    | NULL                  | 4730 | Using where; Using temporary; Using filesort |  |
|  1 | SIMPLE      | v       | eq_ref | PRIMARY,product_id | PRIMARY    | 4       | database.type.id      |    1 | Using where                                  |  |
|  1 | SIMPLE      | hashtag | eq_ref | PRIMARY            | PRIMARY    | 4       | database.v.id         |    1 | Using where                                  |  |
|  1 | SIMPLE      | di      | ref    | product_id         | product_id | 4       | database.v.product_id |  218 | Using where                                  |  |
|  1 | SIMPLE      | docs    | eq_ref | PRIMARY            | PRIMARY    | 4       | database.di.doc_id    |    1 | NULL                                         |  |
+----+-------------+---------+--------+--------------------+------------+---------+-----------------------+------+----------------------------------------------+--+

創建表

Create Table
2Variations CREATE TABLE `2Variations` (
 `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
 `customer_id` int(11) NOT NULL,
 `auto_reorder` tinyint(1) NOT NULL DEFAULT '0',
 `product_id` int(11) NOT NULL,
 `qty_name` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `color` varchar(100) COLLATE utf8mb4_unicode_ci NOT NULL,
 `supplier_code` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `barcode` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `upca` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `no_pic` varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `restock` int(11) NOT NULL DEFAULT '1',
 `hashtag` text COLLATE utf8mb4_unicode_ci,
 `php_wholesale` decimal(11,2) DEFAULT NULL,
 `php_retail` decimal(11,2) DEFAULT NULL,
 `weight` decimal(11,2) NOT NULL DEFAULT '0.00',
 `notes` text COLLATE utf8mb4_unicode_ci,
 `company_notes` text COLLATE utf8mb4_unicode_ci,
 `mgnt_notes` text COLLATE utf8mb4_unicode_ci,
 `deleted` int(11) NOT NULL DEFAULT '0',
 `retired` tinyint(4) NOT NULL DEFAULT '0',
 `created_by` int(20) NOT NULL,
 `created_on` datetime NOT NULL,
 `updated_by` int(20) NOT NULL,
 `last_update` datetime NOT NULL,
 PRIMARY KEY (`id`),
 KEY `product_id` (`product_id`),
 KEY `customer_id` (`customer_id`)
) ENGINE=InnoDB AUTO_INCREMENT=5083 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci



docs    CREATE TABLE `docs` (
     `id` int(11) NOT NULL AUTO_INCREMENT,
     `customer_id` int(11) NOT NULL COMMENT 'id in table customers',
     `registered_id` int(20) NOT NULL DEFAULT '0',
     `doc_type` int(1) NOT NULL DEFAULT '0' COMMENT '0: none, 1:PO, 2:DR, 3:INV, 4: RS',
     `doc_no` varchar(20) COLLATE utf8mb4_unicode_ci NOT NULL,
     `doc_date` date NOT NULL,
     `duedate` date DEFAULT NULL,
     `confirm_date` date NOT NULL,
     `paid` date NOT NULL,
     `paid_amount` decimal(11,2) DEFAULT NULL,
     `sf` decimal(11,2) DEFAULT NULL,
     `rsf` decimal(11,2) DEFAULT NULL,
     `vat` decimal(11,2) DEFAULT NULL,
     `vatinex` int(1) DEFAULT NULL COMMENT '1 = in, 2 = ex',
     `merc_total` decimal(11,2) NOT NULL DEFAULT '0.00',
     `payment` decimal(11,2) NOT NULL DEFAULT '0.00',
     `commission` decimal(11,2) NOT NULL DEFAULT '0.00',
     `discount` decimal(11,2) DEFAULT '0.00',
     `voucher_amount` decimal(11,2) NOT NULL DEFAULT '0.00',
     `bundle_discount_amount` decimal(11,2) NOT NULL DEFAULT '0.00',
     `shipped` datetime DEFAULT NULL,
     `ext_courier` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
     `ext_courier_tracking` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
     `ext_orderid` varchar(50) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
     `ext_total_weight` decimal(11,4) NOT NULL,
     `notes` longtext COLLATE utf8mb4_unicode_ci,
     `company_notes` text COLLATE utf8mb4_unicode_ci,
     `mgnt_notes` longtext COLLATE utf8mb4_unicode_ci,
     `created_by` int(11) NOT NULL COMMENT 'id in users',
     `created_on` datetime NOT NULL,
     `updated_by` int(11) NOT NULL COMMENT 'id in users',
     `last_update` datetime NOT NULL,
     `deleted` int(11) NOT NULL DEFAULT '0',
     `deleted_reason` int(11) NOT NULL DEFAULT '0',
     PRIMARY KEY (`id`),
     KEY `doc_no` (`doc_no`),
     KEY `customer_id` (`customer_id`),
     KEY `ext_courier_tracking` (`ext_courier_tracking`,`ext_orderid`),
     KEY `ext_orderid` (`ext_orderid`)
    ) ENGINE=InnoDB AUTO_INCREMENT=77904 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci



docs_inventory  CREATE TABLE `docs_inventory` (
     `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
     `doc_id` int(11) NOT NULL COMMENT 'id in table doc',
     `product_id` int(11) NOT NULL COMMENT 'id in table products',
     `color` varchar(100) COLLATE utf8mb4_unicode_ci NOT NULL,
     `qty` int(11) NOT NULL DEFAULT '0',
     `warehouse` int(11) NOT NULL DEFAULT '1',
     `price` decimal(11,2) NOT NULL DEFAULT '0.00',
     `sold_price` decimal(11,2) NOT NULL DEFAULT '0.00',
     `cogs` decimal(11,2) DEFAULT NULL,
     `deleted` int(11) NOT NULL DEFAULT '0',
     PRIMARY KEY (`id`),
     KEY `doc_id` (`doc_id`),
     KEY `product_id` (`product_id`)
    ) ENGINE=InnoDB AUTO_INCREMENT=395438 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci



products_type_link  CREATE TABLE `products_type_link` (
     `id` int(11) NOT NULL AUTO_INCREMENT,
     `product_id` int(20) NOT NULL,
     `product_type` int(20) NOT NULL,
     `created_by` int(20) NOT NULL,
     `created_on` datetime NOT NULL,
     `updated_by` int(20) NOT NULL,
     `last_update` datetime NOT NULL,
     `deleted` int(11) NOT NULL DEFAULT '0',
     PRIMARY KEY (`id`),
     KEY `product_type` (`product_type`)
    ) ENGINE=InnoDB AUTO_INCREMENT=4088 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci



variations_hashtag_link CREATE TABLE `variations_hashtag_link` (
     `id` int(11) NOT NULL AUTO_INCREMENT,
     `variation_id` int(20) NOT NULL,
     `variation_hashtag` int(20) NOT NULL,
     `created_by` int(20) NOT NULL,
     `created_on` datetime NOT NULL,
     `updated_by` int(20) NOT NULL,
     `last_update` datetime NOT NULL,
     `deleted` int(11) NOT NULL DEFAULT '0',
     PRIMARY KEY (`id`),
     KEY `variation_hashtag` (`variation_hashtag`)
    ) ENGINE=InnoDB AUTO_INCREMENT=3703 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

UPDATED INDEX 包括 Deleted 變慢到 45 秒

+----+-------------+---------+--------+------------------------------+------------+---------+------------------------+--------+----------------------------------------------+--+
| id | select_type |  table  |  type  |        possible_keys         |    key     | key_len |          ref           |  rows  |                    Extra                     |  |
+----+-------------+---------+--------+------------------------------+------------+---------+------------------------+--------+----------------------------------------------+--+
|  1 | SIMPLE      | di      | ref    | product_id,warehouse,deleted | deleted    |       4 | const                  | 169837 | Using where; Using temporary; Using filesort |  |
|  1 | SIMPLE      | docs    | eq_ref | PRIMARY                      | PRIMARY    |       4 | database.di.doc_id     |      1 | NULL                                         |  |
|  1 | SIMPLE      | v       | ref    | PRIMARY,product_id           | product_id |       4 | database.di.product_id |      2 | Using index                                  |  |
|  1 | SIMPLE      | hashtag | eq_ref | PRIMARY                      | PRIMARY    |       4 | database.v.id          |      1 | Using where                                  |  |
|  1 | SIMPLE      | type    | eq_ref | PRIMARY,deleted,deleted_2    | PRIMARY    |       4 | database.v.id          |      1 | Using where                                  |  |
+----+-------------+---------+--------+------------------------------+------------+---------+------------------------+--------+----------------------------------------------+--+

3 個解決方案

解決方案1
 0  2020-04-04 05:42:01

大規模查詢的另一種替代方法是創建一個臨時表並使用 SQL 作業填充它。 例如，每 10 分鍾執行一次大查詢，將其插入到臨時表中，然后為臨時表編寫查詢。

解決方案2
 0  2020-04-04 06:22:48

不是答案； 評論太長了……

首先，為了提供幫助，我們真的需要查看完整的查詢，而且我個人發現格式化后更容易閱讀，例如如下：

SELECT t.product_type type
     , h.variation_hashtag hashtag
     , di.warehouse
     , SUM(di.qty) qty
     , SUM(di.price) price
     , SUM(di.sold_price) sold_price
     , SUM(di.cogs) cogs
     , AVG(di.price) ave_price
     , AVG(di.sold_price) ave_sold_price
     , YEAR(d.created_on) year
     , MONTH(d.created_on) month
  FROM 2Variations v 
  JOIN docs_inventory di 
    ON di.product_id = v.product_id
  LEFT 
  JOIN docs d
    ON d.id = di.doc_id 
  JOIN variations_type_link t 
    ON t.id = v.id
  LEFT 
  JOIN variations_hashtag_link h
    ON h.id = v.id
 WHERE di.deleted = 0 
   AND t.deleted = 0 
   AND (h.deleted = 0 or h.deleted IS NULL)
   AND di.warehouse BETWEEN 1 AND 2
".$query_where."
 GROUP 
    BY YEAR(d.created_on)
     , MONTH(d.created_on)
     , ".$groups;

請注意，我已經用 INNER JOIN 替換了幾個 OUTER JOIN，因為它們一開始並不是真正的 OUTER JOIN。 此外，在一個名為“id”的列上連接兩個表真的很奇怪。 按照慣例，“id”列代表該表的唯一內容，因此用戶表可能有一個名為“id”的列，而帳戶表可能有一個名為“user_id”的列。

解決方案3
 -1  2020-04-04 04:42:12

對於您的 SQL，以下表格正在工作

  • 2變化=>行：A
  • 變體類型鏈接 => 行：B
  • 變體_hashtag_link => 行：C
  • docs_inventory => 行：D
  • 文檔 => 行：E

因此，如果您的原始查詢將運行，則在 A B C D E 操作之后，連接的工作行和位置將是 A B C D E，group by 將起作用。

提高 SQL 運行速度。 您必須減少操作次數並設置索引鍵。

不用擔心我們可以提高執行時間:)

#1 首先，您必須檢查以下字段是否通過索引設置

  • 2Variations => id, product_id
  • 變體類型鏈接 => id，已刪除
  • 變體_hashtag_link => id，已刪除
  • docs_inventory => product_id, doc_id, 已刪除, 倉庫
  • 文檔 => id

#2 避免在 1 和 2 之間的操作 di.warehouse 之間使用：這不利於快速（倉庫 >= 1 AND 倉庫 <= 2）：有利於快速

#3 嘗試通過使用 sub select 來減少連接操作如果您顯示我的最終查詢，您會有所感覺

我相信下面的查詢會很快工作:)

SELECT dtbl.type, dtbl.hashtag, dtbl.warehouse, sum(dtbl.qty)as qty, sum(dtbl.price) as price, sum(dtbl.sold_price) as sold_price, sum(dtbl.cogs) as cogs, avg(dtbl.price) as ave_price, avg(dtbl.sold_price) as ave_sold_price, 
    YEAR(docs.created_on) as year, MONTH(docs.created_on) as month FROM
(
    SELECT btbl.*, di.* FROM
    (
        SELECT atbl.product_id, atbl.type, hashtag.variation_hashtag as hashtag FROM
        (
            SELECT v.id, v.product_id, type.product_type as type FROM
                2Variations as v
            LEFT JOIN
            (
                SELECT id, product_type FROM variations_type_link
                WHERE deleted = 0
            ) as type
            ON v.id = type.id
        ) as atbl
        LEFT JOIN
        (
            SELECT id, variation_hashtag FROM variations_hashtag_link
            WHERE deleted = 0 OR deleted IS NULL {$query_where}
        ) as hashtag
        ON atbl.id = hashtag.id
    ) as btbl
    LEFT JOIN
    (
        SELECT ctbl.*, YEAR(docs.created_on) as year, MONTH(docs.created_on) as month FROM
        (
            SELECT product_id, qty, warehouse, price, sold_price, cogs, price, sold_price FROM docs_inventory
            WHERE deleted = 0 AND ( warehouse >= 1 AND warehouse <= 2 )
        ) as ctbl
        LEFT JOIN
            docs
        ON ctbl.doc_id = docs.id
    ) as di
    ON btbl.product_id = di.product_id
) as dtbl
GROUP BY YEAR(dtbl.created_on), MONTH(dtbl.created_on), {$groups}

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 MySQL Query選擇了太多行 MySQL查詢返回太多行 MySQL語句拉太多行 MySQL Join返回太多行 MYSQL delete 語句刪除了太多行 同時修改mysql數據庫中的許多行 使用LIKE的PHP MySql搜索匹配太多行 查詢有多個聯接，返回的結果太多 MySQL連接太多 .csv要上傳的行太多？
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM