How can I partition a table on a certain column in MySQL?

Question

I really though this would be simpler than it is. It should be. I'm using MySQL 5.1. I have a table with 3 million rows. Think of it as a table of products. There is a brand_id associated with each product.

brand_id is indexed. The simplest of queries:

select distinct attribute1 from mytable where brand_id in (4,312,122,82,35,313,123,83,360,170,36,314,124,84,361,171,172,37,315,125,85,362,38,316,126,86,363,173,39,317,127,87,364,174,318,128,365,175,319,129,88,366,176,89,367,177,368,178,369,179,420,230,421,231,422,232,470,233,280,424,471,234,281,425,472,235,282,426,473,236,283,427,474,237,284,428,475,238,10,285,429,476,239,286,477,11,287,478,60,12,100,288,479,61,13,101,289,62,14,102,63,340,150,15,103,64,341,151,16,104,65,342,152,17,105,343,153,18,106,390,66,344,154,19,107,391,67,345,155,108,392,68,346,156,109,393,69,347,157,394,348,158,395,349,159,396,397,400,210,398,401,211,399,402,212,450,260,403,213,451,261,404,214,452,262,405,215,453,263,406,216,454,264,407,217,455,265,408,218,409,456,266,1,219,457,267,2,458,268,3,40,459,269,4,41,5,42,90,6,43,320,130,91,7,321,131,92,8,44,322,132,93,9,370,180,45,323,133,94,371,181,46,324,134,95,372,182,47,325,135,96,373,183,48,326,136,97,374,184,49,327,137,98,375,185,328,138,376,186,329,139,99,377,187,378,188,379,189,430,240,431,241,432,242,433,480,290,243,434,481,291,244,435,482,292,245,436,483,293,246,437,484,294,247,438,485,295,248,20,439,486,296,249,21,487,297,488,300,110,298,70,22,489,301,111,299,71,23,302,112,72,24,113,73,350,160,25,303,114,74,351,161,26,304,75,352,162,27,305,115,76,353,163,28,306,116,354,164,29,307,117,77,355,165,308,118,78,356,166,309,119,79,357,167,358,168,359,169,410,220,411,221,412,222,413,460,270,223,414,461,271,224,415,462,272,225,416,463,273,226,417,464,274,227,418,465,275,228,419,466,276,229,467,277,468,278,50,469,279,51,52,140,53,330,54,331,141,332,142,380,190,55,333,143,381,191,56,334,144,382,192,57,335,145,193,383,58,336,146,194,384,59,337,147,195,385,338,148,196,386,339,149,197,387,200,388,198,201,389,19)

is taking 4 seconds. I have adjusted all kinds of MySQL settings and changed from InnoDB to MyISAM. Still amazing takes 4 seconds. Incredible.

So I thought I would partition on brand_id. I thought, why not try it?

alter table mytable partition by key(brand_id);

and I am met with:

ERROR 1503 (HY000): A PRIMARY KEY must include all columns in the table's partitioning function

I don't know what that means, unfortunately. All I REALLY want is for this query to go faster than 4 (4!) seconds. What can I do to solve this problem, and why can't MySQL perform this very simple task?

Answer 1

Stick your set of ids in a temp table, then join that table to mytable.

CREATE TEMPORARY TABLE brand_ids (brand_id int);

INSERT INTO brand_ids (brand_id)
VALUES (4), (312), (122), ...;

SELECT DISTINCT mytable.attribute1
FROM mytable
  JOIN brand_ids
    ON mytable.brand_id = brand_ids.brand_id;

Answer 2

To answer your question about partitioning, the partition function for a table MUST include the table's primary key. So, unless you make brand_id part of your primary key, you can't partition based on that column.

As to the more general question of poor performance with large IN clauses, might want to have a look at this question