MySql query runs very slow(actually never gives output) without where clause

Question

I have a mysql query and it works fine when i use where clause, but when i donot use where clause it gone and never gives the output and finally timeout. Actually i have used Explain command to check the performance of the query and in both cases the Explain gives the same number of rows used in joining. I have attached the image of output got with Explain command. Below is the query. I couldn't figure whats the problem here. Any help is highly appreciated. Thanks.

 SELECT 
        MCI.CLIENT_ID AS CLIENT_ID, MCI.NAME AS CLIENT_NAME, MCI.PRIMARY_CONTACT AS CLIENT_PRIMARY_CONTACT, 
        MCI.ADDED_BY AS SP_ID, CONCAT(MUD_SP.FIRST_NAME, ' ', MUD_SP.LAST_NAME) AS SP_NAME, 
        MCI.FK_PROSPECT_ID AS PROSPECT_ID, MCI.DATE_ADDED AS ADDED_ON,
        (SELECT GROUP_CONCAT(LT.TAG_TEXT SEPARATOR ', ') 
            FROM LK_TAG LT
            INNER JOIN M_OBJECT_TAG_MAPPING MOTM
            ON LT.PK_ID = MOTM.FK_TAG_ID
            WHERE MOTM.FK_OBJECT_ID = MCI.FK_PROSPECT_ID
                AND MOTM.OBJECT_TYPE = 1
                AND MOTM.IS_ACTIVE = 1
        ) AS TAGS,
        IFNULL(SUM(GET_DIGITS(MMR.RCP_AMOUNT)), 0) AS REVENUE_SO_FAR,
        IFNULL(SUM(GET_DIGITS(MMR.RCP_RUPEES)), 0) AS REVENUE_INR,
        COUNT(DISTINCT PMI_MONTHLY.PROJECT_ID) AS MONTHLY,
        COUNT(DISTINCT PMI_FIXED.PROJECT_ID) AS FIXED,
        COUNT(DISTINCT PMI_HOURLY.PROJECT_ID) AS HOURLY,
        COUNT(DISTINCT PMI_ANNUAL.PROJECT_ID) AS ANNUAL,
        COUNT(DISTINCT PMI_CURRENTLY_RUNNING.PROJECT_ID) AS CURRENTLY_RUNNING_PROJECTS,
        COUNT(DISTINCT PMI_YET_TO_START.PROJECT_ID) AS YET_TO_START_PROJECTS,
        COUNT(DISTINCT PMI_TECH_SALES_CLOSED.PROJECT_ID) AS TECH_SALES_CLOSED_PROJECTS
    FROM 
        M_CLIENT_INFO MCI
        INNER JOIN M_USER_DETAILS MUD_SP
            ON MCI.ADDED_BY = MUD_SP.PK_ID
        LEFT OUTER JOIN M_MONTH_RECEIPT MMR
                ON MMR.CLIENT_ID = MCI.CLIENT_ID
        LEFT OUTER JOIN M_PROJECT_INFO PMI_FIXED
                ON PMI_FIXED.CLIENT_ID =  MCI.CLIENT_ID AND PMI_FIXED.PROJECT_TYPE = 1
        LEFT OUTER JOIN M_PROJECT_INFO PMI_MONTHLY
                ON PMI_MONTHLY.CLIENT_ID =  MCI.CLIENT_ID AND PMI_MONTHLY.PROJECT_TYPE = 2
        LEFT OUTER JOIN M_PROJECT_INFO PMI_HOURLY
                ON PMI_HOURLY.CLIENT_ID =  MCI.CLIENT_ID AND PMI_HOURLY.PROJECT_TYPE = 3
        LEFT OUTER JOIN M_PROJECT_INFO PMI_ANNUAL
                ON PMI_ANNUAL.CLIENT_ID =  MCI.CLIENT_ID AND PMI_ANNUAL.PROJECT_TYPE = 4
        LEFT OUTER JOIN M_PROJECT_INFO PMI_CURRENTLY_RUNNING
                ON PMI_CURRENTLY_RUNNING.CLIENT_ID =  MCI.CLIENT_ID AND PMI_CURRENTLY_RUNNING.STATUS = 4
        LEFT OUTER JOIN M_PROJECT_INFO PMI_YET_TO_START
                ON PMI_YET_TO_START.CLIENT_ID =  MCI.CLIENT_ID AND PMI_YET_TO_START.STATUS < 4
        LEFT OUTER JOIN M_PROJECT_INFO PMI_TECH_SALES_CLOSED
                ON PMI_TECH_SALES_CLOSED.CLIENT_ID =  MCI.CLIENT_ID AND PMI_TECH_SALES_CLOSED.STATUS > 4
               WHERE YEAR(MCI.DATE_ADDED) = '2012'
                GROUP BY MCI.CLIENT_ID ORDER BY CLIENT_NAME ASC  

Answer 1

Yes, as many people have said, the key is that when you have the where clause, mysql engine filters the table M_CLIENT_INFO --probably drammatically--.

A similar result as removing the where clause is to to add this where clause:

where 1 = 1

You will see that the performance is degraded also because mysql will try to get all the data.

Answer 2

Remove the where clause and all columns from select and add a count to see how many records you get. If it is reasonable, say up to 10k, then do the following,

1. put back the select columns related to M_CLIENT_INFO

2. do not include the nested one "TAGS"

3. remove all your joins

4. run your query without where clause and gradually include the joins

this way you'll find out when the timeout is caused.

Answer 3

I would try the following. First, MySQL has a keyword "STRAIGHT_JOIN" which tells the optimizer to do the query in the table order you've specified. Since all you left-joins are child-related (like a lookup table), you don't want MySQL to try and interpret one of those as a primary basis of the query.

SELECT STRAIGHT_JOIN ... rest of query.

Next, your M_PROJECT_INFO table, I dont know how many columns of data are out there, but you appear to be concentrating on just a few columns on your DISTINCT aggregates. I would make sure you have a covering index on these elements to help the query via an index on

( Client_ID, Project_Type, Status, Project_ID )

This way the engine can apply the criteria and get the distinct all out of the index instead of having to go back to the raw data pages for the query.

Third, your M_CLIENT_INFO table. Ensure that has an index on both your criteria, group by AND your Order By, and change your order by from the aliased "CLIENT_NAME" to the actual column of the SQL table so it matches the index

( Date_Added, Client_ID, Name )

I have "name" in ticks as it is also a reserved word and helps clarify the column, not the keyword.

Next, the WHERE clause. Whenever you apply a function to an indexed column name, it doesn't work the greatest, especially on date/time fields... You might want to change your where clause to

WHERE MCI.Date_Added between '2012-01-01' and '2012-12-31 23:59:59'

so the BETWEEN range is showing the entire year and the index can better be utilized.

Finally, if the above do not help, I would consider splitting your query some. The GROUP_CONCACT inline select for the TAGS might be a bit of a killer for you. You might want to have all the distinct elements first for the grouping per client, THEN get those details.... Something like

select
      PQ.*,
      group_concat(...) tags
   from
      ( the entire primary part of the query ) as PQ
         Left join yourGroupConcatTableBasis on key columns