大表联接的mysql查询优化

Question

I am creating a report for radio station which generates logs of online listeners to keep records of ip, date, time, total user listening etc.

Listeners Table

client_ip        date        time      date_time            listeners  
---------------  ----------  --------  -------------------  -----------
166.147.81.179   2012-04-30  00:19:46  2012-04-30 00:19:46            1
64.12.243.203    2012-04-30  04:38:37  2012-04-30 04:38:37            1
198.228.211.195  2012-04-30  05:36:33  2012-04-30 05:36:33            1
198.228.211.195  2012-04-30  05:36:34  2012-04-30 05:36:34            2
198.228.211.195  2012-04-30  05:36:35  2012-04-30 05:36:35            2
198.228.211.195  2012-04-30  05:36:35  2012-04-30 05:36:35            3
166.147.81.179   2012-04-30  05:47:13  2012-04-30 05:47:13            2
76.170.251.97    2012-04-30  06:01:37  2012-04-30 06:01:37            2
76.170.251.97    2012-04-30  06:01:39  2012-04-30 06:01:39            2
76.170.251.97    2012-04-30  06:01:39  2012-04-30 06:01:39            2

At the same time it keeps records of song details (title, artist, album, lenght, date, time) etc.

Playlists Table

title                       artist                           length_in_secs  played_date  played_time  start_date_time      end_date_time        
--------------------------  -------------------------------  --------------  -----------  -----------  -------------------  ---------------------
We Found Love               Rihanna                                     184  2012-04-30   00:00:21     2012-04-30 00:00:21  2012-04-30 00:03:25  
Photograph                  Nickelback                                  216  2012-04-30   00:03:31     2012-04-30 00:03:31  2012-04-30 00:07:07  
Not Over You                Gavin DeGraw                                214  2012-04-30   00:07:18     2012-04-30 00:07:18  2012-04-30 00:10:52  
Stereo Hearts               Gym Class Heroes Ft Adam Levine             210  2012-04-30   00:10:55     2012-04-30 00:10:55  2012-04-30 00:14:25  
I Gotta Feeling             Black  Eyed Peas                            243  2012-04-30   00:15:03     2012-04-30 00:15:03  2012-04-30 00:19:06  
One Thing Leads To Another  Fixx                                        182  2012-04-30   00:19:14     2012-04-30 00:19:14  2012-04-30 00:22:16  
Raise Your Glass            Pink                                        202  2012-04-30   00:22:29     2012-04-30 00:22:29  2012-04-30 00:25:51  
Better In Time              Leona Lewis                                 216  2012-04-30   00:30:13     2012-04-30 00:30:13  2012-04-30 00:33:49  
Tainted Love                Soft Cell                                   153  2012-04-30   00:33:56     2012-04-30 00:33:56  2012-04-30 00:36:29  
Haven't Met You Yet         Michael Buble'                              242  2012-04-30   00:37:14     2012-04-30 00:37:14  2012-04-30 00:41:16  

So, the report requirement is "how many user listen the song within the date or in date range" and I write the query like this. It gives the output correct (as far as I know) but query execution takes time disproportional to data size - from 5 seconds to 5-10 minutes, depending on date range.

SELECT DATE_FORMAT(p.played_date, "%m/%d/%Y") `played_date`, p.played_time, p.length_in_secs, p.title, p.artist, RTRIM(p.label) `label`, RTRIM(p.album) `album`, IFNULL((SELECT SUM(l.listeners) FROM listeners `l` WHERE l.date_time >= p.start_date_time AND l.date_time <= p.end_date_time LIMIT 1), 0) `listeners` FROM playlists `p` WHERE p.title <> "" AND (p.played_date >= '2012-04-30' AND p.played_date <= '2012-05-30') HAVING listeners > 0 ORDER BY p.title ASC;
// formatted //
SELECT 
    DATE_FORMAT(p.played_date, "%m/%d/%Y") `played_date`,
    p.played_time,
    p.length_in_secs,
    p.title,
    p.artist,
    RTRIM(p.label) `label`,
    RTRIM(p.album) `album`,
    IFNULL(
        (SELECT 
            SUM(l.listeners) 
        FROM
            listeners `l` 
        WHERE l.date_time >= p.start_date_time 
            AND l.date_time <= p.end_date_time 
        LIMIT 1),
        0
    ) `listeners` 
FROM
    playlists `p` 
WHERE p.title <> "" 
    AND (
        p.played_date >= '2012-04-30' 
        AND p.played_date <= '2012-05-30'
    ) 
HAVING listeners > 0 
ORDER BY p.title ASC

Output:

played_date  played_time  length_in_secs  title                  artist                    label               album               listeners  
-----------  -----------  --------------  ---------------------  ------------------------  ------------------  ------------------  -----------
04/30/2012   08:06:26                228  Brighter Than The Sun  Colbie Caillat (Cal-Lay)  Universal Republic  All of You                    9

04/30/2012   08:44:16                248  Breakfast At Tiffanys  Deep Blue Something                                                         6

04/30/2012   18:06:40                253  Bizarre Love Triangle  New Order                                                                   2

04/30/2012   17:05:21                183  Animal                 Neon Trees                Mercury             Habits                        5

04/30/2012   08:58:05                253  Always Be My Baby      Mariah Carey                                                                2

04/30/2012   07:25:52                264  Already Gone           Kelly Clarkson            RCA                 All I Ever Wante              3

04/30/2012   16:21:33                236  All The Right Moves    One Republic              Interscope          Waking Up                     7

04/30/2012   11:58:26                199  All That She Wants     Ace Of Base                                                                12

04/30/2012   11:14:17                247  All I Wanna Do         Sheryl Crow                                                                 2

04/30/2012   16:12:59                235  A Thousand Miles       Vanessa Carlton                                                             5

Is there a way to optimize this query to run faster, or write a new, faster one? Please suggest/help me. Thank you!!

Using EXPLAIN

EXPLAIN playlists;


Field            Type              Null    Key     Default            Extra                        
---------------  ----------------  ------  ------  -----------------  -----------------------------
playlist_id      int(10) unsigned  NO      PRI     (NULL)             auto_increment               
title            varchar(255)      YES             (NULL)                                          
artist           varchar(255)      YES             (NULL)                                          
label            varchar(255)      YES             (NULL)                                          
album            varchar(255)      YES             (NULL)                                          
length_in_secs   int(11)           NO              (NULL)                                          
played_date      date              NO              (NULL)                                          
played_time      time              NO              (NULL)                                          
start_date_time  datetime          NO              (NULL)                                          
end_date_time    datetime          NO              (NULL)                                          
added_date       datetime          NO              (NULL)                                          
modified_date    timestamp         NO              CURRENT_TIMESTAMP  on update CURRENT_TIMESTAMP


EXPLAIN listeners;


Field          Type                 Null    Key     Default            Extra                        
-------------  -------------------  ------  ------  -----------------  -----------------------------
listener_id    bigint(20) unsigned  NO      PRI     (NULL)             auto_increment               
station_id     int(10) unsigned     NO              (NULL)                                          
client_ip      varchar(50)          NO              (NULL)                                          
time           time                 NO              (NULL)                                          
date           date                 NO              (NULL)                                          
date_time      datetime             YES             (NULL)                                          
timestamp      bigint(20) unsigned  NO              (NULL)                                          
listeners      int(10) unsigned     NO              (NULL)                                          
processes      int(10) unsigned     NO              (NULL)                                          
uid            int(10) unsigned     NO              (NULL)                                          
user_agent     varchar(255)         YES             (NULL)                                          
added_date     datetime             NO              (NULL)                                          
modified_date  timestamp            NO              CURRENT_TIMESTAMP  on update CURRENT_TIMESTAMP  

Answer 1

Use INNER JOIN instead of using correlated subquery as:

SELECT DATE_FORMAT(p.played_date, "%m/%d/%Y") played_date,
       p.played_time,
       p.length_in_secs,
       p.title,
       p.artist,
       RTRIM(p.label) label,
       RTRIM(p.album) album,
       SUM(l.listeners) listeners
FROM playlists p
     INNER JOIN listeners l
         ON l.date_time BETWEEN p.start_date_time AND p.end_date_time
WHERE p.title <> "" AND
      p.played_date BETWEEN '2012-04-30' AND  '2012-05-30'
ORDER BY p.title ASC;

Consider adding following indexes on tables may help you improve performance of a query. Check for the following indexes with EXPLAIN :

playlists KEY (played_date, start_date_time, end_date_time, title);

listeners KEY (date_time, listeners);

Answer 2

As discussed in the comments, your query doesn't actually do what you want it to do. Given the data you have, I would personally process this outside of SQL to create a table to store how many people listened to each song, which you can then query in SQL to get this information. If you absolutely want an SQL query to do this however, it will need to be something along the lines of this monstrocity;

SELECT 
DATE_FORMAT(p.played_date, "%m/%d/%Y") `played_date`,
p.played_time,
p.length_in_secs,
p.title,
p.artist,
RTRIM(p.label) `label`,
RTRIM(p.album) `album`,
SUM(SMALLEST(prev_listeners,next_listeners,dur_listeners) AS listeners
FROM (
  SELECT
  P.start_date_time,
  SUBSTRING_INDEX(GROUP_CONCAT(l_before.listeners ORDER BY l_before.date_time DESC),',',1) AS prev_listeners, 
  SUBSTRING_INDEX(GROUP_CONCAT(l_after.listeners ORDER BY l_after.date_time ASC),',',1) AS next_listeners, 
  MIN(l_during) AS dur_listeners
  FROM playlists p
  JOIN listeners l_before ON l_before.date_time < p.start_date_time 
  LEFT JOIN listeners l_after ON l_before.client_ip = l_after.client_ip AND l_after.date_time > p.end_date_time 
  LEFT JOIN listeners l_during ON l.client_ip = l_during.client_ip AND l_during.date_time BETWEEN p.start_date_time AND p.end_date_time
  WHERE p.title <> ""
  AND p.played_date BETWEEN '2012-04-30' AND '2012-05-30'
  GROUP BY p.start_date_time, l_before.client_ip
) l
JOIN playlists p USING (start_date_time)
GROUP BY p.start_date_time
ORDER BY p.start_date_time

Where SMALLEST is a function that returns the smallest non_null argument.

This will take considerably longer than your current query, but it's the most efficient way I can think of for getting the actual number of listeners for each song.

Oh, and this is assuming that the log records a row with zero listeners when everyone from an ip address stops listening, otherwise there's really no way to do this.