MySQL query running very slow on Composite key table

Question

I've got a composite key table CUSTOMER_PRODUCT_XREF

__________________________________________________________________
|CUSTOMER_ID (PK NN VARCHAR(191)) | PRODUCT_ID(PK NN VARCHAR(191))|
-------------------------------------------------------------------

In my batch program I need to select 500 updated customers and also get the PRODUCT_ID's purchased by CUSTOMERs separated by comma and update our SOLR index. In my query I'm select 500 customers and doing a left join to CUSTOMER_PRODUCT_XREF

SELECT 
    customer.*, group_concat(xref.PRODUCT_ID separator ', ')
FROM
    CUSTOMER customer
LEFT JOIN CUSTOMER_PRODUCT_XREF xref ON customer.CUSTOMER_ID=xref.CUSTOMER_ID  
group by customer.CUSTOMER_ID 
LIMIT 500;

EDIT: EXPLAIN QUERY

id  select_type table   type    possible_keys   key      key_len    ref     rows    Extra
1   SIMPLE     customer ALL      PRIMARY        NULL     NULL       NULL    74236   Using where; Using temporary; Using filesort
1   SIMPLE      xref    index     NULL          PRIMARY  1532       NULL    121627  Using where; Using index; Using join buffer (Block Nested Loop)

I got lost connection exception after 20 minutes running the above query.

I tried with the following (sub query) and it took 1.7 seconds to get result but still slow.

SELECT 
customer.*, (SELECT group_concat(PRODUCT_ID separator ', ') 
     FROM CUSTOMER_PRODUCT_XREF xref 
        WHERE customer.CUSTOMER_ID=xref.CUSTOMER_ID
        GROUP BY customer.CUSTOMER_ID) 
FROM
CUSTOMER customer
LIMIT 500;

EDIT: EXPLAIN QUERY produces

id  select_type          table      type    possible_keys   key    key_len  ref     rows   Extra
1   PRIMARY              customer     ALL       NULL        NULL    NULL    NULL    74236   NULL
2   DEPENDENT SUBQUERY    xref      index        NULL     PRIMARY   1532    NULL    121627 Using where; Using index; Using temporary; Using filesort

Question

CUSTOMER_PRODUCT_XREF already has both columns set as PRIMARY_KEY and NOT_NULL but why is my query still very slow ? I thought having Primary Key on a column was enough to build an index for it. Do I need further indexing ?

DATABASE INFO:

• All the ID's in my database are VARCHAR(191) because the id's can contain alphabets.
• I'm using utf8mb4_unicode_ci character encoding
• I'm using SET group_concat_max_len := @@max_allowed_packet to get maximum number of product_ids for each customer. Prefer using group_concat in one main query so that I don't have to do multiple separate queries to get products for each customer.

Answer 1

Your original version of the query is doing the join first and then sorting all the resulting data -- which is probably pretty big given how large the fields are.

You can "fix" that version by selecting 500 hundred customers first and then doing the join:

SELECT c.*, group_concat(xref.PRODUCT_ID separator ', ')
FROM (select c.*
      from CUSTOMER customer c
      order by c.customer_id
      limit 500
     ) c LEFT JOIN
     CUSTOMER_PRODUCT_XREF xref
     ON c.CUSTOMER_ID=xref.CUSTOMER_ID  
group by c.CUSTOMER_ID ;

An alternative that might or might not have a big impact would be to doing the aggregation by customer in a subquery and join that, as in:

SELECT c.*, xref.products
FROM (select c.*
      from CUSTOMER customer c
      order by c.customer_id
      limit 500
     ) c LEFT JOIN
     (select customer_id, group_concat(xref.PRODUCT_ID separator ', ') as products
      from CUSTOMER_PRODUCT_XREF xref
     ) xref
     ON c.CUSTOMER_ID=xref.CUSTOMER_ID;

What you have discovered is that the MySQL optimizer does not recognize this situation (where the limit has a big impact on performance). Some other database engines do a better job of optimization in this case.

Answer 2

Alright the speed of the queries in my question shot up when I created an index just on the CUSTOMER_ID in CUSTOMER_PRODUCT_XREF table.

So I've got two indexes now PRIMARY_KEY_INDEX on PRODUCT_ID and CUSTOMER_ID CUSTOMER_ID_INDEX on CUSTOMER_ID