MySQL Slow query ~ 10 seconds

Question

I have this query which basically goes through a bunch of tables to get me some formatted results but I can't seem to find the bottleneck. The easiest bottleneck was the ORDER BY RAND() but the performance are still bad.

The query takes from 10 sec to 20 secs without ORDER BY RAND() ;

SELECT 
  c.prix AS prix, 
  ST_X(a.point) AS X, 
  ST_Y(a.point) AS Y, 
  s.sizeFormat AS size, 
  es.name AS estateSize, 
  c.title AS title, 
  DATE_FORMAT(c.datePub, '%m-%d-%y') AS datePub,
  dbr.name AS dateBuiltRange, 
  m.myId AS meuble, 
  c.rawData_id AS rawData_id, 
  GROUP_CONCAT(img.captionWebPath) AS paths 
FROM 
 immobilier_ad_blank AS c 
 LEFT JOIN PropertyFeature AS pf ON (c.propertyFeature_id = pf.id) 
 LEFT JOIN Adresse AS a ON (c.adresse_id = a.id) 
 LEFT JOIN Size AS s ON (pf.size_id = s.id) 
 LEFT JOIN EstateSize AS es ON (pf.estateSize_id = es.id) 
 LEFT JOIN Meuble AS m ON (pf.meuble_id = m.id) 
 LEFT JOIN DateBuiltRange AS dbr ON (pf.dateBuiltRange_id = dbr.id) 
 LEFT JOIN ImageAd AS img ON (img.commonAd_id = c.rawData_id) 
WHERE 
  c.prix != 0
  AND pf.subCatMyId = 1 
  AND (
    (
      c.datePub > STR_TO_DATE('01-04-2016', '%d-%m-%Y') 
      AND c.datePub < STR_TO_DATE('30-04-2016', '%d-%m-%Y')
    ) 
    OR date_format(c.datePub, '%d-%m-%Y') = '30-04-2016'
 ) 
AND a.validPoint = 1 
GROUP BY 
  c.id 
#ORDER BY 
#  RAND() 
LIMIT 
 5000

Here is the explain query:

Visual Portion:

And here is a screenshot of mysqltuner

EDIT 1

I have many indexes Here they are:

EDIT 2:

So you guys did it. Down to .5 secs to 2.5 secs.

I mostly followed all of your advices and changed some of my.cnf + runned optimized on my tables.

Answer 1

You're searching for dates in a very suboptimal way. Try this.

 ...  c.datePub >= STR_TO_DATE('01-04-2016', '%d-%m-%Y') 
  AND c.datePub < STR_TO_DATE('30-04-2016', '%d-%m-%Y') + INTERVAL 1 DAY

That allows a range scan on an index on the datePub column. You should create a compound index for that table on (datePub, prix, addresse_id, rawData_id) and see if it helps.

Also try an index on a (valid_point) . Notice that your use of a geometry data type in that table is probably not helping anything.

Answer 2

To begin with you have quite a lot of indexes but many of them are not useful. Remember more indexes means slower inserts and updates. Also mysql is not good at using more than one index per table in complex queries. The following indexes have a cardinality < 10 and probably should be dropped.

IDX_...E88B
IDX....62AF
IDX....7DEE
idx2
UNIQ...F210
UNIQ...F210..
IDX....0C00
IDX....A2F1
At this point I got tired of the excercise, there are many more

Then you have some duplicated data.

point lat lng

The point field has the lat and lng in it. So the latter two are not needed. That means you can lose two more indexes idxlat and idxlng . I am not quite sure how idxlng appears twice in the index list for the same table.

These optimizations will lead to an overall increase in performance for INSERTS and UPDATES and possibly for all SELECTs as well because the query planner needs to spend less time deciding which index to use.

Then we notice from your explain that the query does not use any index on table Adresse (a). But your where clause has a.validPoint = 1 clearly you need an index on it as suggested by @Ollie-Jones

However I suspect that this index may have low cardinality. In that case I recommend that you create a composite index on this column + another.

Answer 3

The problem is your join with (a). The table has an index, but the index can't be used, more than likely due to the sort (/group by), or possibly incompatible types. The EXPLAIN shows three quarters of a million rows examined, this means that index lookup was not possible.

When designing a query, look for the smallest possible result set - search by that index, and then join from there. Perhaps "c" isn't the best table for the primary query.

(You could try using FORCE INDEX (id) on table a, if it doesn't work, the error may give you more information).

Answer 4

As others have pointed out, you need an index on a.validPoint but what about c.datePub that is also used in the WHERE clause. Why not a multiple column index on datePub, address_id the index on address_id is already used, so a multiple column index will be better here.