Slow count query with where clause

Question

I am trying to perform a count to get the total number of results in a pagination but the query is too slow 2.12s

+-------+
| size  |
+-------+
| 50000 |
+-------+
1 row in set (2.12 sec)

my count query

select  count(appeloffre0_.ID_APPEL_OFFRE) as size  
from  ao.appel_offre appeloffre0_  
inner join ao.acheteur acheteur1_ 
on appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR 
where 
(exists (select 1 from ao.lot lot2_ where lot2_.ID_APPEL_OFFRE=appeloffre0_.ID_APPEL_OFFRE and lot2_.ESTIMATION_COUT>=1)) 
and (exists (select 1 from ao.lieu_execution lieuexecut3_ where lieuexecut3_.appel_offre=appeloffre0_.ID_APPEL_OFFRE and lieuexecut3_.region=1)) 
and (exists (select 1 from ao.ao_activite aoactivite4_ where aoactivite4_.ID_APPEL_OFFRE=appeloffre0_.ID_APPEL_OFFRE and (aoactivite4_.ID_ACTIVITE=1))) 
and appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01' 
and (appeloffre0_.CATEGORIE='fournitures' or appeloffre0_.CATEGORIE='travaux' or appeloffre0_.CATEGORIE='services') 
and acheteur1_.ID_ENTITE_MERE=2

explain cmd :

+----+--------------------+--------------+------+---------------------------------------------+--------------------+---------+--------------------------------+-------+--------------------------+
| id | select_type        | table        | type | possible_keys                               | key                | key_len | ref                            | rows  | Extra                    |
+----+--------------------+--------------+------+---------------------------------------------+--------------------+---------+--------------------------------+-------+--------------------------+
|  1 | PRIMARY            | acheteur1_   | ref  | PRIMARY,acheteur_ibfk_1                     | acheteur_ibfk_1    | 5       | const                          |     3 | Using where; Using index |
|  1 | PRIMARY            | appeloffre0_ | ref  | appel_offre_ibfk_2                          | appel_offre_ibfk_2 | 4       | ao.acheteur1_.ID_ACHETEUR      | 31061 | Using where              |
|  4 | DEPENDENT SUBQUERY | aoactivite4_ | ref  | ao_activites_activite_fk,ao_activites_ao_fk | ao_activites_ao_fk | 4       | ao.appeloffre0_.ID_APPEL_OFFRE |     3 | Using where              |
|  3 | DEPENDENT SUBQUERY | lieuexecut3_ | ref  | fk_ao_lieuex,fk_region_lieuex               | fk_ao_lieuex       | 4       | ao.appeloffre0_.ID_APPEL_OFFRE |     1 | Using where              |
|  2 | DEPENDENT SUBQUERY | lot2_        | ref  | FK_LOT_AO                                   | FK_LOT_AO          | 4       | ao.appeloffre0_.ID_APPEL_OFFRE |     5 | Using where              |
+----+--------------------+--------------+------+---------------------------------------------+--------------------+---------+--------------------------------+-------+--------------------------+

the index acheteur_ibfk_1 is a FK references table ENTITE_MERE because i have and acheteur1_.ID_ENTITE_MERE=2 in where clause.

Answer 1

You can have multiple conditions on your joins by using ON condition1 AND condition2 etc.

SELECT COUNT(appeloffre0_.ID_APPEL_OFFRE) as size  
FROM  ao.appel_offre appeloffre0_  
JOIN ao.acheteur acheteur1_ ON appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR 
JOIN ao.lot lot2_ ON appeloffre0_.ID_APPEL_OFFRE=lot2_.ID_APPEL_OFFRE AND lot2_.ESTIMATION_COUT>=1 
JOIN ao.lieu_execution lieuexecut3_ ON appeloffre0_.ID_APPEL_OFFRE=lieuexecut3_.ID_APPEL_OFFRE AND lieuexecut3_.ID_ACTIVITE=1
JOIN ao.ao_activite aoactivite4_ ON appeloffre0_.ID_APPEL_OFFRE=aoactivite4_.ID_APPEL_OFFRE AND aoactivite4_.ID_ACTIVITE=1
WHERE appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01' 
AND (appeloffre0_.CATEGORIE='fournitures' OR appeloffre0_.CATEGORIE='travaux' OR appeloffre0_.CATEGORIE='services') 
AND acheteur1_.ID_ENTITE_MERE=2;

Answer 2

You can try:

select count(aa.ID_APPEL_OFFRE) as size  
from  (
select ID_APPEL_OFFRE, ID_ACHETEUR from ao.appel_offre appeloffre0_  
inner join ao.acheteur acheteur1_ 
on appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR 
where appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01' 
and (appeloffre0_.CATEGORIE in ('fournitures','travaux','services')) 
and (acheteur1_.ID_ENTITE_MERE=2)) aa
inner join ao.lot lot2_ on lot2_.ID_APPEL_OFFRE=aa.ID_APPEL_OFFRE
inner join ao.lieu_execution lieuexecut3_ on lieuexecut3_.appel_offre=aa.ID_APPEL_OFFRE
inner join ao.ao_activite aoactivite4_  on aoactivite4_.ID_APPEL_OFFRE=aa.ID_APPEL_OFFRE
where 
aoactivite4_.ID_ACTIVITE=1
and lot2_.ESTIMATION_COUT>=1
and lieuexecut3_.region=1;

But I haven't seen your tables so I am not 100% sure that you won't get duplicates because of joins.

A couple of low-hanging fruits might also be found by ensuring that your appeloffre0_.CATEGORIE and appeloffre0_.DATE_OUVERTURE_PLIS have indexes on them.

Other fields which should have indexes on them are ao.lot.ID_APPEL_OFFRE, ao.lieu_execution.ID_APPEL_OFFRE and ao.ao_activite.ID_APPEL_OFFRE, and ao.appel_offre.ID_ACHETEUR (all the joined fields).

Answer 3

I would have the following indexes on your tables if not already... These are covering indexes for your query meaning the index has the applicable column to get your results without having to go to the actual raw data pages.

table           index
appel_offre     ( DATE_OUVERTURE_PLIS, CATEGORIE, ID_APPEL_OFFRE, ID_ACHETEUR )
lot             ( ID_APPEL_OFFRE, ESTIMATION_COUT )
lieu_execution  ( appel_offre, region )
ao_activite     ( ID_APPEL_OFFRE, ID_ACTIVITE )

Having indexes on just individual columns won't really help optimize what you are looking for. Also, I am doing count of DISTINCT ID_APPEL_OFFRE's in case any of the JOINed tables have more than 1 record, it does not create a Cartesian result count for you

select  
      count(distinct AOF.ID_APPEL_OFFRE) as size
   from
      ao.appel_offre AOF
         JOIN ao.acheteur ACH
            on AOF.ID_ACHETEUR = ACH.ID_ACHETEUR 
           and ACH.ID_ENTITE_MERE = 2
         JOIN ao.lot
            ON AOF.ID_APPEL_OFFRE = lot.ID_APPEL_OFFRE
           and lot.ESTIMATION_COUT >= 1
         JOIN ao.lieu_execution EX
            ON AOF.ID_APPEL_OFFRE = EX.appel_offre
            and EX.region = 1
         JOIN ao.ao_activite ACT
            ON AOF.ID_APPEL_OFFRE = ACT.ID_APPEL_OFFRE 
           and ACT.ID_ACTIVITE = 1
   where 
          AOF.DATE_OUVERTURE_PLIS > '2015-01-01'
      and (   AOF.CATEGORIE = 'fournitures' 
           or AOF.CATEGORIE = 'travaux' 
           or AOF.CATEGORIE = 'services') 

Answer 4

Like @FuzzyTree said in his comment exists is faster than an inner join if it's not a 1:1 relationship because it terminates as soon as it finds 1 whereas the join will get every matching row.

But the solution is that We add in and not exists :

where ( appeloffre0_.ID_APPEL_OFFRE IN (select lot2_.ID_APPEL_OFFRE from ao.lot lot2_  
             where    lot2_.ESTIMATION_COUT>=1)
      ) 

So the query run very fast than exists or joins .

select  count(appeloffre0_.ID_APPEL_OFFRE) as size  
from  ao.appel_offre appeloffre0_  
inner join ao.acheteur acheteur1_ 
on appeloffre0_.ID_ACHETEUR=acheteur1_.ID_ACHETEUR 
where 
( appeloffre0_.ID_APPEL_OFFRE IN (select lot2_.ID_APPEL_OFFRE from ao.lot lot2_  where    lot2_.ESTIMATION_COUT>=1)) 
and (appeloffre0_.ID_APPEL_OFFRE IN (select lieuexecut3_.appel_offre from ao.lieu_execution lieuexecut3_  where  lieuexecut3_.region=1)) 
and (appeloffre0_.ID_APPEL_OFFRE IN (select aoactivite4_.ID_APPEL_OFFRE from ao.ao_activite aoactivite4_  where aoactivite4_.ID_ACTIVITE=1 )) 
and appeloffre0_.DATE_OUVERTURE_PLIS>'2015-01-01' 
and (appeloffre0_.CATEGORIE='fournitures' or appeloffre0_.CATEGORIE='travaux' or appeloffre0_.CATEGORIE='services') 
and acheteur1_.ID_ENTITE_MERE=2