Slow query when large number of returned columns

Question

I have a fairly large query that is taking about 0.11 seconds to execute from within a php script (not massive, but might be fairly heavily used), but when executed through phpmyadmin it claims the query is taking 0.0047 seconds.

I have done a fair bit of investigation and I have found that if I reduce the returned columns to a minimum (from ~50 down to 2, leaving those used to sort the results) the query is far faster (within the same region as phpmyadmin claims). Unfortunately I need the data in those columns. If the returned data is exported it is only around 3k (as a CSV). Removing the order by clause also has negligible effect.

The number of rows returned is quite small (about 8), so the limit clause that phpmyadmin uses by default is not a factor.

It seems strange that the amount of data retrieved is having such a dramatic effect on the query (rather than on the fetch of any row of data).

I am unsure what I can do to improve performance without reducing the returned columns, or splitting the query up (which would just change the bottleneck).

The query is as follows (not sure how much help this is with the apparent problem).

SELECT DISTINCT
    f.MultiItemService,
    b.GroupName,
    n.CourierName,
    f.DeliveryserviceId,
    f.CourierId,
    f.DeliveryserviceName,
    f.CustomerDescription,
    f.SageCode,
    d.MarkupId,
    d.Description AS MarkupDescription,
    d.Discount,
    d.FixedAmount,
    f.Status,
    f.MinItems,
    f.MaxItems,
    f.MinWeight,
    f.MaxWeight,
    f.MinVolume,
    f.MaxVolume,
    f.MinWidth,
    f.MaxWidth,
    f.MinHeight,
    f.MaxHeight,
    f.MinDepth,
    f.MaxDepth,
    f.MinWorth,
    f.MaxWorth,
    d.MinItems AS MarkupMinItems,
    d.MaxItems AS MarkupMaxItems,
    d.MinWeight AS MarkupMinWeight,
    d.MaxWeight AS MarkupMaxWeight,
    d.MinVolume AS MarkupMinVolume,
    d.MaxVolume AS MarkupMaxVolume,
    d.MinWidth AS MarkupMinWidth,
    d.MaxWidth AS MarkupMaxWidth,
    d.MinHeight AS MarkupMinHeight,
    d.MaxHeight AS MarkupMaxHeight,
    d.MinDepth AS MarkupMinDepth,
    d.MaxDepth AS MarkupMaxDepth,
    d.MinWorth AS MarkupMinWorth,
    d.MaxWorth AS MarkupMaxWorth,
    f.Price,
    f.AdditionalPrice,
    f.DeliveryType,
    f.Consolidation,
    f.Surcharge,
    h.VatRate,
    k.InsuranceLevelId,
    k.InsuranceLevelDescription,
    k.InsuranceLevelMinimum,
    k.InsuranceLevelMaximum,
    m.OptionId, 
    m.OptionDescription, 
    m.OptionCost,
    f.CalculationId,
    CASE b.GroupName  WHEN 'home' THEN 1   WHEN 'customer' THEN 2  WHEN 'Standard' THEN 3 ELSE 4 END AS Priority
FROM users a
INNER JOIN groups b ON a.UserId = b.UserId AND a.ApiKey = 'ABC123ABC' 
INNER JOIN markups d ON b.GroupId = d.GroupId
INNER JOIN markups_deliveryservices o ON d.MarkupId = o.MarkupId 
INNER JOIN deliveryservices f ON o.DeliveryserviceId = f.DeliveryserviceId
INNER JOIN deliveryservices_days j ON f.DeliveryserviceId = j.DeliveryserviceId 
INNER JOIN couriers n ON f.CourierId = n.CourierId  
LEFT OUTER JOIN insurance_levels k ON f.DeliveryserviceId = k.DeliveryserviceId 
LEFT OUTER JOIN optional_services_deliveryservices l ON f.DeliveryserviceId = l.DeliveryserviceId 
LEFT OUTER JOIN optional_services m ON l.OptionId = m.OptionId  
INNER JOIN deliveryservices_delivery_groups g ON f.DeliveryserviceId = g.DeliveryserviceId
INNER JOIN delivery_groups h ON g.DeliveryGroupId = h.DeliveryGroupId 
INNER JOIN markups_delivery_groups p ON d.MarkupId = p.MarkupId
INNER JOIN delivery_groups q ON p.DeliveryGroupId = q.DeliveryGroupId 
WHERE h.DeliveryGroupId IN (15,12)
AND q.DeliveryGroupId IN (15,12) AND f.DeliveryType = 'Business And Domestic'
AND a.Status = 1
AND b.Status = 1
AND d.Status = 1
AND n.Status = 1
AND f.Status = 1
AND 1 BETWEEN f.MinItems AND f.MaxItems 
AND ((f.MultiItemService = 0 /* single parcel delivery services */
AND 32 BETWEEN f.MinWeight AND f.MaxWeight  /* item total weight within limits */
AND 193960 BETWEEN f.MinVolume AND f.MaxVolume  /* item total volume within limits */
AND 0 BETWEEN f.MinWorth AND f.MaxWorth)  /* item total value within limits */
OR (f.MultiItemService = 1 /* multiple parcel delivery services */
AND 0.16 BETWEEN f.MinWeight AND f.MaxWeight   /* max weight of any item within limits */
AND 969.8 BETWEEN f.MinVolume AND f.MaxVolume   /* max volume of any item within limits */
AND 0 BETWEEN f.MinWorth AND f.MaxWorth   /* max value of any item within limits */
AND ((32 != 0 AND 32 NOT BETWEEN f.MinWeight AND f.MaxWeight)  /* item total weight NOT within limits */
OR (149200 != 0 AND 193960 NOT BETWEEN f.MinVolume AND f.MaxVolume)  /* item total volume NOT within limits */
OR (0 != 0 AND 0 NOT BETWEEN f.MinWorth AND f.MaxWorth)
)))
AND 18.4 BETWEEN f.MinWidth AND f.MaxWidth 
AND 15.6 BETWEEN f.MinHeight AND f.MaxHeight 
AND 2.6 BETWEEN f.MinDepth AND f.MaxDepth
AND 1 BETWEEN d.MinItems AND d.MaxItems 
AND 32 BETWEEN d.MinWeight AND d.MaxWeight 
AND 193960 BETWEEN d.MinVolume AND d.MaxVolume 
AND 18.4 BETWEEN d.MinWidth AND d.MaxWidth 
AND 15.6 BETWEEN d.MinHeight AND d.MaxHeight 
AND 2.6 BETWEEN d.MinDepth AND d.MaxDepth
AND 0 BETWEEN d.MinWorth AND d.MaxWorth
AND j.DayOfWeek = 5 
AND j.UpToTime >= 61063
ORDER BY Priority

And the explain is as follows:-

id  select_type     table   type    possible_keys   key     key_len     ref     rows    Extra 
1   SIMPLE  k   system  DeliveryserviceId   NULL    NULL    NULL    0   const row not found
1   SIMPLE  l   system  RuleId  NULL    NULL    NULL    0   const row not found
1   SIMPLE  m   system  PRIMARY NULL    NULL    NULL    0   const row not found
1   SIMPLE  a   ref PRIMARY,ApiKey  ApiKey  257 const   1   Using where; Using temporary; Using filesort
1   SIMPLE  q   index   PRIMARY PRIMARY 4   NULL    7   Using where; Using index; Using join buffer
1   SIMPLE  p   ref PRIMARY,DeliveryGroupId,MarkupId    DeliveryGroupId 4   DeliveryApiAdv.q.DeliveryGroupId    2    
1   SIMPLE  d   eq_ref  PRIMARY,GroupId PRIMARY 4   DeliveryApiAdv.p.MarkupId   1   Using where
1   SIMPLE  b   eq_ref  PRIMARY,UserId  PRIMARY 4   DeliveryApiAdv.d.GroupId    1   Using where
1   SIMPLE  o   ref markup_id,DeliveryserviceId markup_id   4   DeliveryApiAdv.p.MarkupId   2   Using index
1   SIMPLE  g   ref PRIMARY,DeliveryGroupId,DeliveryserviceId   PRIMARY 4   DeliveryApiAdv.o.DeliveryserviceId  1   Using where; Using index
1   SIMPLE  h   eq_ref  PRIMARY PRIMARY 4   DeliveryApiAdv.g.DeliveryGroupId    1    
1   SIMPLE  j   eq_ref  PRIMARY PRIMARY 5   DeliveryApiAdv.o.DeliveryserviceId,const    1   Using where
1   SIMPLE  f   eq_ref  PRIMARY,CourierId   PRIMARY 4   DeliveryApiAdv.o.DeliveryserviceId  1   Using where
1   SIMPLE  n   eq_ref  PRIMARY PRIMARY 4   DeliveryApiAdv.f.CourierId  1   Using where

EDIT - Bit of a follow up after more investigation. Cutting down the number of columns returned reduces the run time dramatically by about 95%. The point where this happens varies on what columns are used (ie, about a dozen INT columns, but only about 8 reasonable size VARCHAR columns). For an experiment I ran the SQL using CONCAT_WS to join the returned columns together rather than being individually returned. This increased the number of columns worth of data returned from 11 to 22.

For some further debugging I have changed the main SELECT so it just returns the ID fields of the other tables, but with no change to the joins at all. I have then used this to populate a temp table. This runs rapidly. I have then joined that temp table back against the tables to get the data I actually need. This query has very simple joins on the primary keys and no where clause, but runs slowly.

To me this confirms the problem is not the complexity of the query, rather some limit I am hitting related to the resulting row size.

EDIT - after further investigation the problem does not appear to be down to the query. Running it on a slightly differently configured database results in this not being an issue.

Answer 1

You may try to use some part of where clause in the join condition.

eg

select a.*, b.* from a inner join b on a.col1 = b.col1 and b.status = 1 and a.status = 1