Mysql subquery much faster than join

Question

I have the following queries which both return the same result and row count:

select * from (
               select UNIX_TIMESTAMP(network_time) * 1000 as epoch_network_datetime, 
                      hbrl.business_rule_id, 
                      display_advertiser_id, 
                      hbrl.campaign_id, 
                      truncate(sum(coalesce(hbrl.ad_spend_network, 0))/100000.0, 2) as demand_ad_spend_network, 
                      sum(coalesce(hbrl.ad_view, 0)) as demand_ad_view, 
                      sum(coalesce(hbrl.ad_click, 0)) as demand_ad_click, 
                      truncate(coalesce(case when sum(hbrl.ad_view) = 0 then 0 else 100*sum(hbrl.ad_click)/sum(hbrl.ad_view) end, 0), 2) as ctr_percent, 
                      truncate(coalesce(case when sum(hbrl.ad_view) = 0 then 0 else sum(hbrl.ad_spend_network)/100.0/sum(hbrl.ad_view) end, 0), 2) as ecpm,
                      truncate(coalesce(case when sum(hbrl.ad_click) = 0 then 0 else sum(hbrl.ad_spend_network)/100000.0/sum(hbrl.ad_click) end, 0), 2) as ecpc 
               from hourly_business_rule_level hbrl
               where (publisher_network_id = 31534) 
               and network_time between str_to_date('2017-08-13 17:00:00.000000', '%Y-%m-%d %H:%i:%S.%f') and str_to_date('2017-08-14 16:59:59.999000', '%Y-%m-%d %H:%i:%S.%f') 
               and (network_time IS NOT NULL and display_advertiser_id > 0)
               group by network_time, hbrl.campaign_id, hbrl.business_rule_id
               having demand_ad_spend_network > 0
               OR demand_ad_view > 0
               OR demand_ad_click > 0
               OR ctr_percent > 0
               OR ecpm > 0
               OR ecpc > 0
               order by epoch_network_datetime) as atb
       left join dim_demand demand on atb.display_advertiser_id = demand.advertiser_dsp_id 
       and atb.campaign_id = demand.campaign_id 
       and atb.business_rule_id = demand.business_rule_id 

ran explain extended, and these are the results:

+----+-------------+----------------------------+------+-------------------------------------------------------------------------------+---------+---------+-----------------+---------+----------+----------------------------------------------+
| id | select_type | table                      | type | possible_keys                                                                 | key     | key_len | ref             | rows    | filtered | Extra                                        |
+----+-------------+----------------------------+------+-------------------------------------------------------------------------------+---------+---------+-----------------+---------+----------+----------------------------------------------+
|  1 | PRIMARY     | <derived2>                 | ALL  | NULL                                                                          | NULL    | NULL    | NULL            | 1451739 |   100.00 | NULL                                         |
|  1 | PRIMARY     | demand                     | ref  | PRIMARY,join_index                                                            | PRIMARY | 4       | atb.campaign_id |       1 |   100.00 | Using where                                  |
|  2 | DERIVED     | hourly_business_rule_level | ALL  | _hourly_business_rule_level_supply_idx,_hourly_business_rule_level_demand_idx | NULL    | NULL    | NULL            | 1494447 |    97.14 | Using where; Using temporary; Using filesort |
+----+-------------+----------------------------+------+-------------------------------------------------------------------------------+---------+---------+-----------------+---------+----------+----------------------------------------------+

and the other is:

select UNIX_TIMESTAMP(network_time) * 1000 as epoch_network_datetime, 
       hbrl.business_rule_id, 
       display_advertiser_id, 
       hbrl.campaign_id, 
       truncate(sum(coalesce(hbrl.ad_spend_network, 0))/100000.0, 2) as demand_ad_spend_network, 
       sum(coalesce(hbrl.ad_view, 0)) as demand_ad_view, 
       sum(coalesce(hbrl.ad_click, 0)) as demand_ad_click, 
       truncate(coalesce(case when sum(hbrl.ad_view) = 0 then 0 else 100*sum(hbrl.ad_click)/sum(hbrl.ad_view) end, 0), 2) as ctr_percent, 
       truncate(coalesce(case when sum(hbrl.ad_view) = 0 then 0 else sum(hbrl.ad_spend_network)/100.0/sum(hbrl.ad_view) end, 0), 2) as ecpm, 
       truncate(coalesce(case when sum(hbrl.ad_click) = 0 then 0 else sum(hbrl.ad_spend_network)/100000.0/sum(hbrl.ad_click) end, 0), 2) as ecpc 
from hourly_business_rule_level hbrl
join dim_demand demand on hbrl.display_advertiser_id = demand.advertiser_dsp_id 
and hbrl.campaign_id = demand.campaign_id 
and hbrl.business_rule_id = demand.business_rule_id 
where (publisher_network_id = 31534) 
and network_time between str_to_date('2017-08-13 17:00:00.000000', '%Y-%m-%d %H:%i:%S.%f') and str_to_date('2017-08-14 16:59:59.999000', '%Y-%m-%d %H:%i:%S.%f') 
and (network_time IS NOT NULL and display_advertiser_id > 0)
group by network_time, hbrl.campaign_id, hbrl.business_rule_id
having demand_ad_spend_network > 0
OR demand_ad_view > 0
OR demand_ad_click > 0 
OR ctr_percent > 0
OR ecpm > 0
OR ecpc > 0
order by epoch_network_datetime;

and these are the results for the second query:

+----+-------------+----------------------------+------+-------------------------------------------------------------------------------+---------+---------+---------------------------------------------------------------+---------+----------+----------------------------------------------+
| id | select_type | table                      | type | possible_keys                                                                 | key     | key_len | ref                                                           | rows    | filtered | Extra                                        |
+----+-------------+----------------------------+------+-------------------------------------------------------------------------------+---------+---------+---------------------------------------------------------------+---------+----------+----------------------------------------------+
|  1 | SIMPLE      | hourly_business_rule_level | ALL  | _hourly_business_rule_level_supply_idx,_hourly_business_rule_level_demand_idx | NULL    | NULL    | NULL                                                          | 1494447 |    97.14 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | demand                     | ref  | PRIMARY,join_index                                                            | PRIMARY | 4       | my6sense_datawarehouse.hourly_business_rule_level.campaign_id |       1 |   100.00 | Using where; Using index                     |
+----+-------------+----------------------------+------+-------------------------------------------------------------------------------+---------+---------+---------------------------------------------------------------+---------+----------+----------------------------------------------+

the first one takes about 2 seconds while the second one takes over 2 minutes!

why is the second query taking so long? what am I missing here?

thanks.

Answer 1

One possible reason is the number of rows that have to be joined with the second table.

The GROUP BY clause and the HAVING clause will limit the number of rows returned from your subquery. Only those rows will be used for the join.

Without the subquery only the WHERE clause is limiting the number of rows for the JOIN. The JOIN is done before the GROUP BY and HAVING clauses are processed. Depending on group size and the selectivity of the HAVING conditions there would be much more rows that need to be joined.

Consider the following simplified example:

We have a table users with 1000 entries and the columns id , email .

create table users(
    id smallint auto_increment primary key,
    email varchar(50) unique
);

Then we have a (huge) log table user_actions with 1,000,000 entries and the columns id , user_id , timestamp , action

create table user_actions(
    id mediumint auto_increment primary key,
    user_id smallint not null,
    timestamp timestamp,
    action varchar(50),
    index (timestamp, user_id)
);

The task is to find all users who have at least 900 entries in the log table since 2017-02-01.

The subquery solution:

select a.user_id, a.cnt, u.email
from (
    select a.user_id, count(*) as cnt
    from user_actions a
    where a.timestamp >= '2017-02-01 00:00:00'
    group by a.user_id
    having cnt >= 900
) a
left join users u on u.id = a.user_id

The subquery returns 135 rows (users). Only those rows will be joined with the users table. The subquery runs in about 0.375 seconds. The time needed for the join is almost zero, so the full query runs in about 0.375 seconds.

Solution without subquery:

select a.user_id, count(*) as cnt, u.email
from user_actions a
left join users u on u.id = a.user_id
where a.timestamp >= '2017-02-01 00:00:00'
group by a.user_id
having cnt >= 900

The WHERE condition filters the table to 866,081 rows. The JOIN has to be done for all those 866K rows. After the JOIN the GROUP BY and the HAVING clauses are processed and limit the result to 135 rows. This query needs about 0.815 seconds.

So you can already see, that a subquery can improve the performance.

But let's make things worse and drop the primary key in the users table. This way we have no index which can be used for the JOIN. Now the first query runs in 0.455 seconds. The second query needs 40 seconds - almost 100 times slower .

Notes

It's difficult to say if the same applies to your case. Reasons are:

• Your queries are quite complex and far away from from beeing an MVCE .
• I don't see anything beeng selected from the demand table - So it's unclear why you are joining it at all.
• You use a LEFT JOIN in one query and an INNER JOIN in another one.
• The relation between the two tables is unclear.
• No information about indexes. You should provide the CREATE statements ( SHOW CREATE table_name ).

Test setup

drop table if exists users;
create table users(
    id smallint auto_increment primary key,
    email varchar(50) unique
)
    select seq as id, rand(1) as email
    from seq_1_to_1000
;


drop table if exists user_actions;
create table user_actions(
    id mediumint auto_increment primary key,
    user_id smallint not null,
    timestamp timestamp,
    action varchar(50),
    index (timestamp, user_id)
)
    select seq as id
        , floor(rand(2)*1000)+1 as user_id
        #, '2017-01-01 00:00:00' + interval seq*20 second as timestamp
        , from_unixtime(unix_timestamp('2017-01-01 00:00:00') + seq*20) as timestamp
        , rand(3) as action
    from seq_1_to_1000000
;

MariaDB 10.0.19 with sequence plugin.

Answer 2

The queries are different. One says JOIN , the other says LEFT JOIN . You are not using demand , so the join is probably useless. However, in the case of JOIN , you are filtering out advertisers that are not in dim_demand ; it that the intent?

But that does not address the question.

The EXPLAINs estimate that there are 1.5M rows in hbrl . But how many show up in the result? I would guess it is a lot fewer. From this, I can answer your question.

Consider these two:

SELECT ... FROM ( SELECT ... FROM a
                      GROUP BY or HAVING or LIMIT ) x
           JOIN b

SELECT ... FROM a
           JOIN b
           GROUP BY or HAVING or LIMIT

The first will decrease the number of rows that need to join to b ; the second will need to do a full 1.5M joins. I suspect that the time taken to do the JOIN (be it LEFT or not) is where the difference is.

Plan A: Remove demand from the query.

Plan B: Use a subquery whenever the subquery significantly shrinks the number of rows before the JOIN .

Indexing (may speed up both variants):

INDEX(publisher_network_id, network_time)

and get rid of this as being useless (since the between will fail anyway for NULL ):

and network_time IS NOT NULL

Side note: I recommend simplifying and fixing this

and  network_time
   between str_to_date('2017-08-13 17:00:00.000000', '%Y-%m-%d %H:%i:%S.%f')
       AND str_to_date('2017-08-14 16:59:59.999000', '%Y-%m-%d %H:%i:%S.%f')

to

and network_time >= '2017-08-13 17:00:00
and network_time  < '2017-08-13 17:00:00 + INTERVAL 24 HOUR

Answer 3

Use a subquery whenever the subquery significantly shrinks the number of rows before - ANY JOIN - always to reinforce Rick James Plan B. To reinforce Rick & Paul's answer which you have already documented. The answers by Rick and Paul deserve Acceptance.