优化选择计数（DISTINCT ip）

Question

I'm trying to get aggregated results (total unique IPs) from a table with about 2M new rows every day.

The table:

CREATE TABLE `clicks` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `hash` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `type` enum('popunder','gallery','exit','direct') COLLATE utf8_unicode_ci NOT NULL,
  `impression_time` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `source_user_id` int(11) NOT NULL,
  `destination_user_id` int(11) NOT NULL,
  `destination_campaign_id` int(11) NOT NULL,
  `destination_campaign_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `destination_campaign_url` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `ip` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `referrer` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `country_code` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `country_id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `country` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `isp` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
  `category_id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `category` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
  `bid` float(8,2) NOT NULL,
  `created_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `updated_at` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  PRIMARY KEY (`id`),
  KEY `ip` (`ip`),
  KEY `source_user_id` (`source_user_id`),
  KEY `destination_user_id` (`destination_user_id`),
  KEY `destination_campaign_id` (`destination_campaign_id`),
  KEY `clicks_hash_index` (`hash`),
  KEY `clicks_created_at_index` (`created_at`),
  KEY `campaign_date` (`destination_campaign_id`,`created_at`),
  KEY `source_user_date` (`source_user_id`,`created_at`)
) ENGINE=InnoDB AUTO_INCREMENT=301539660 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

My query:

SELECT SUM(ips_by_date.count) as count, ips_by_date.date as date
FROM (SELECT count(DISTINCT ip) as count, DATE(created_at) as date 
      FROM clicks as clicks 
      WHERE created_at BETWEEN '2016-05-22 00:00:00' AND '2016-05-23 23:59:59' 
      GROUP BY DATE(created_at)) as ips_by_date 
GROUP BY date;

Now, this query took 93 seconds to run for just one day and I feel like I'm missing something.

Is there any optimization I can make to speed up the performance of this simple count?

Thank you.

Answer 1

First, I don't see why a subquery is necessary. The inner query has one row per date. There is no need to aggregate again. Second, your query is for two days, but I get the points about performance.

So, let's start with:

SELECT count(DISTINCT ip) as count, DATE(created_at) as date 
FROM clicks  
WHERE created_at BETWEEN '2016-05-22 00:00:00' AND '2016-05-23 23:59:59' 
GROUP BY DATE(created_at);

For this query, you want an index on clicks(created_at, ip) . Note also that I would write this as:

SELECT count(DISTINCT ip) as count, DATE(created_at) as date 
FROM clicks  
WHERE created_at >= '2016-05-22' AND created_at < '2016-05-24' 
GROUP BY DATE(created_at);

This should show some improvement, but I don't think it will be radically better because a file sort is still necessary for the outer aggregation.

Answer 2

The performance here boils down to the efficiency of your indexes, since there is not much room for changes in your code (see Gordons code for a cleaner version of your code).

An index on (created_at) or (created_at, ip) will unfortunatley not directly give you distinct ip without further sorting (since you don't group by created_at ), but the latter at least would not require direct table access. So the next optimization would require an index on (date(created_at), ip) , even though that would mean some duplication of data.

As of mysql 5.7.6, you can use a generated column to create a column dt as date(created_at) , before 5.7.6, just create a column dt and update it manually (if you ever change your create_at -value, you have to add a trigger to update that column accordingly). Your initial update might take a while, so update in batches or consider just using it for future querys.

Adding an index (dt, ip) should now give you the result with a single index/range scan and without filesort and without the need to calculate date() from datetime:

select count(distinct ip) as count, dt 
from clicks  
where dt >= '2016-05-22' and dt < '2016-05-24' 
group by dt;

If everything works fine, this should take you just some seconds even for some million rows.

Some things that still might cause you troubles: Since 90 seconds is still a relatively big number for 2 million rows, it might indicate that you have issues with buffer sizes / ram / hdd. If it takes you eg 80 seconds to rebuffer and load your index into memory, there is not much an index can do after that. An easy test for this: run your query twice. If it is (really) significantly faster the second time (like << 1/10th), then you might have to think about tweaking your system settings, architecture or partitioning. Having said that, you should not tweak your system (and sometimes not even add another index or a date column) for query like this and maybe slow down other, more important things - to get daily statistics, you could just as easily run a task at midnight for all statistics you can think of and save the results for you to look at in the morning nice and easy, it would not matter if it takes hours for your query to run.

Answer 3

First add the composite index already mentioned. Then the real performance problem will be reading a zillion rows to compute a COUNT(DISTINCT...) . That action requires either collecting all the values, sorting and doing a GROUP BY , or trying to keep all the distinct values in RAM.

Summary tables are wonderful for speeding up SUM , COUNT , and even AVG in Data Warehousing applications. But COUNT(DISTINCT...) (aka "count unique users") does not lend itself to Summary tables. If you are willing to accept a small error, there is a way. See my blog .

You may not realize it, but the blanket use of 255 in VARCHAR sometimes causes unnecessary performance problems. In this case, you have ip taking 765 bytes in any tmp table, perhaps in the query in question. Changing it to VARCHAR(39) CHARACTER SET ascii would cut that back by a factor of 20! (It is hard to predict how much, if any, that will speed up your query. You could get it down to BINARY(16) with a simple stored function.