MySQL，DISTINCT在SUM操作中

Question

Currently I trying to calculate number of unique user visit in my application based on user gender. Here is the example query that calculate all the visits (not unique)

SELECT
    DATE(v.visited_at) AS visit_date,
    SUM(IF(u.gender = 'M', 1, 0)) AS male_visit,
    SUM(IF(u.gender = 'F', 1, 0)) AS female_visit,
    SUM(IF(u.gender = '' OR u.gender IS NULL, 1, 0)) AS unknown_visit
FROM 
    visits v
    INNER JOIN users u ON v.user_id = u.id
WHERE
    DATE(v.visited_at) >= DATE_SUB(SYSDATE(), INTERVAL 30 DAY)
    AND v.duration > 30
GROUP BY
    DATE(v.visited_at)

Tried using subquery and count distinct it's works, but it's 4 times slower.

SELECT
    DATE(visited_at) as visit_date,
    (SELECT COUNT(DISTINCT u.id) FROM visits v JOIN users u ON v.user_id = u.id WHERE u.gender = 'M' AND DATE(v.visited_at) = visit_date AND v.duration > 30) AS male_visit,
    (SELECT COUNT(DISTINCT u.id) FROM visits v JOIN users u ON v.user_id = u.id WHERE u.gender = 'F' AND DATE(v.visited_at) = visit_date AND v.duration > 30) AS female_visit,
    (SELECT COUNT(DISTINCT u.id) FROM visits v JOIN users u ON v.user_id = u.id WHERE u.gender = '' OR u.gender IS NULL AND DATE(v.visited_at) = visit_date AND v.duration > 30) AS unknown_visit
FROM 
    visits v
WHERE
    DATE(visited_at) >= DATE_SUB(SYSDATE(), INTERVAL 30 DAY)
GROUP BY
    DATE(visited_at)

Any suggestion on this?

Answer 1

COUNT(DISTINCT) is always going to be slower than COUNT() . You can try:

SELECT DATE(v.visited_at) AS visit_date,
       COUNT(DISTINCT CASE WHEN u.gender = 'M' THEN u.id END) AS male_visit,
       COUNT(DISTINCT CASE WHEN u.gender = 'F' THEN u.id END) AS female_visit,
       COUNT(DISTINCT CASE WHEN u.gender = '' OR u.gender IS NULL THEN u.id END) AS unknown_visit
FROM visits v INNER JOIN
     users u
     ON v.user_id = u.id
WHERE DATE(v.visited_at) >= DATE_SUB(SYSDATE(), INTERVAL 30 DAY) AND
      v.duration > 30
GROUP BY DATE(v.visited_at);

I don't know if it will be much faster, though.

Answer 2

There are 2 tables as per query (user and visit) with sample data.

Query

SELECT
DATE(v.visited_date) AS visit_date,
u.gender,
COUNT(DISTINCT v.user_id) AS total_count
FROM
visits v
INNER JOIN users u ON v.user_id = u.id
WHERE
DATE(v.visited_date) >= DATE_SUB(SYSDATE(), INTERVAL 30 DAY)
AND v.duration >= 30
GROUP BY u.gender,DATE(v.visited_date)
ORDER BY DATE(v.visited_date) ASC;

---

This query will give you unique count of users gender wise for particular date.

Answer 3

This type of query is likely to be slow, especially if you have a large number of entries in the table as when selecting rows based upon date and time values mysql has to perform a full table scan.

Optimising your database structure is likely to offer you performance gains much in excess of anything you will get trying to query it like this.

A couple of suggestions would be to partition the table by date ranges. Doing so can greatly reduce query execution as it means instead of a full table scan mysql can simply ignore any partitions outside the query date range. The bigger the table the more benefit you will see, but potentially anything from 2x to 10x faster I would expect.

If you were to replace your gender column with 3 columns male , female and unknown you would replace 3 queries containing the slow COUNT(DISTINCT... statements with a single query with less conditions, you can also add the user id to the group by statement to remove the need to count distinct as you can specify more than one column for grouping.

Finally you could add a database trigger and either have an extra column which it sets as 1 when logging the visits if the duration is over 30 and it's their first visit of the day, or you create a new calendar table for visits and have the trigger increment the value within that upon database write of each log which equates to a unique visit for the day.