Poor UNION ALL performance in MySQL

Question

I have a database with rows like the following:

+------------+---------+------------+-------+
| continent  | country | city       | value |
+------------+---------+------------+-------+
| Asia       | China   | Beijing    | 3     |
| ...        | ...     | ...        | ...   |
| N. America | USA     | D.C        | 7     |
| ....       | ....    | ....       | ....  |

In order to generate a treemap visualization, I need to work this into a table with the following shape:

+-----+------------+-------+
| uid | parent-uid | value |
+-----+------------+-------+

In this case, Asia is the "parent" for China , which is the "parent" for Beijing . So for those three you'd have something like:

+---------+--------+-----+
| Beijing | China  | 3   |
| China   | Asia   | ... |
| Asia    | global | ... |
+---------+--------+-----+

The "value" for China needs to be an aggregate of all child values. Similarly the value of Asia needs to be an aggregate of all child values.

To accomplish this purely in SQL I created the following three queries and combined them with UNION ALL :

# City-level:
SELECT
     CONCAT(continent, "-", country, "-", city) as uid,
     CONCAT(continent, "-", country) as parentuid,
     value
FROM
     table

UNION ALL

# Country-level
SELECT
     CONCAT(continent, "-", country) as uid,
     continent as parentuid,
     SUM(value) as value
FROM
     table
GROUP BY
     country

UNION ALL

# Continent-level
SELECT
    continent as uid,
    "global" as parentuid,
    SUM(value) as value
FROM
    table
GROUP BY
    continent

Each of the individual queries completes in milliseconds. City-level, country-level, and continent-level all return results in < 0.01 seconds

When I union them all together it suddenly takes 8 seconds to get results!

I've tried Googling the issues but everything just says "Use UNION ALL instead of UNION " (I already am)

I considered that it might not have enough RAM to build the temporary results table so it's disk trashing, but I don't know how to increase the memory limit. I tried bumping innodb_buffer_pool_size to 1GB (1073741824) but it didn't help

Answer 1

The first select , selects all rows in the table then getting the first row is very fast but fetching all rows will take very much time(the mysql workbench append limit 1000 to end of the query by default).

To test that fetching all rows take more time, try following query and say us the time it consumes:

select * from (
SELECT
     CONCAT(continent, "-", country, "-", city) as uid,
     CONCAT(continent, "-", country) as parentuid,
     value
FROM
     table
) t1;

If it takes almost 8 seconds then your union have no problem. And for improve performance you must limit rows by using where clause.

I hope it could help.

Answer 2

I guess my question is: what's wrong with WITH ROLLUP ?

SELECT
    CONCAT_WS('-',continent,country,city) as uid,
    CONCAT_WS('-',continent,COALESCE(country,'global')) as parentuid,
    value
FROM (
    SELECT continent, country, city, SUM(value) as value
    FROM table
    GROUP BY continent, country, city WITH ROLLUP
) t1
WHERE t1.continent IS NOT NULL;

I may not have the CONCAT_WS() calls correct, especially if you have cities or countries named '' , but I have to think this would be faster. The WHERE clause is just there to remove the overall summary.

Here's the example for WITH ROLLUP from the MySQL doc to help explain what it does:

mysql> SELECT year, country, product, SUM(profit)
    -> FROM sales
    -> GROUP BY year, country, product WITH ROLLUP;
+------+---------+------------+-------------+
| year | country | product    | SUM(profit) |
+------+---------+------------+-------------+
| 2000 | Finland | Computer   |        1500 |
| 2000 | Finland | Phone      |         100 |
| 2000 | Finland | NULL       |        1600 |
| 2000 | India   | Calculator |         150 |
| 2000 | India   | Computer   |        1200 |
| 2000 | India   | NULL       |        1350 |
| 2000 | USA     | Calculator |          75 |
| 2000 | USA     | Computer   |        1500 |
| 2000 | USA     | NULL       |        1575 |
| 2000 | NULL    | NULL       |        4525 |
| 2001 | Finland | Phone      |          10 |
| 2001 | Finland | NULL       |          10 |
| 2001 | USA     | Calculator |          50 |
| 2001 | USA     | Computer   |        2700 |
| 2001 | USA     | TV         |         250 |
| 2001 | USA     | NULL       |        3000 |
| 2001 | NULL    | NULL       |        3010 |
| NULL | NULL    | NULL       |        7535 |
+------+---------+------------+-------------+