optimizing a query using union left joins and inner joins

Question

I have a query that I want to optimize. Both unions executed separatley run fine, however as soon as I include inner join it takes up to 57 seconds. How do I solve this. My query is as follows

SELECT
    p.PROJID,
    p.StartDate,
    o.ORDERNO,
    p.PROJCODE,
    p.PROJECT,
    cat.type AS CATEGORY,
    p.AREA,
    p.STATE,
    p.COUNTRY,
    p.VALUE,
    p.PROCESSOR,
    p.PROJINFO,
    p.NES,
    p.SPECSALE,
    p.OFFICE,
    p.DEPTCODE,
    p.INTERNLCHG,
    p.INTERCOCHG,
    p.LORM,
    p.PERCENT,
    d.COMPANY,
    CONCAT(
        d.LASTNAME,
        ", ",
        d.FIRSTNAME
    ) AS Contact
FROM
    (
        (
            SELECT
                *
            FROM
                (
                    SELECT
                        `clients`.`CLIENTID` AS `CLIENTIDA`,
                        `clients`.`COMPANY` AS `COMPANY`
                    FROM
                        `hdb`.`clients`
                    UNION
                        SELECT
                            `accounts`.`id` AS `CLIENTIDA`,
                            `accounts`.`name` AS `COMPANY`
                        FROM
                            `sugarcrm`.`accounts`
                ) AS a
            INNER JOIN (
                SELECT
                    `hdb`.`contacts`.`CONTACTID` AS `CONTACTID`,
                    `hdb`.`contacts`.`CLIENTID` AS `CLIENTIDC`,
                    `hdb`.`contacts`.`FIRSTNAME` AS `FIRSTNAME`,
                    `hdb`.`contacts`.`LASTNAME` AS `LASTNAME`
                FROM
                    `hdb`.`contacts`
                UNION
                    SELECT
                        `sugarcrm`.`contacts`.`id` AS `CONTACTID`,
                        `sugarcrm`.`accounts_contacts`.`account_id` AS `CLIENTIDC`,
                        `sugarcrm`.`contacts`.`first_name` AS `FIRSTNAME`,
                        `sugarcrm`.`contacts`.`last_name` AS `LASTNAME`
                    FROM
                        `sugarcrm`.`contacts`
                    LEFT JOIN `sugarcrm`.`accounts_contacts` ON `sugarcrm`.`contacts`.`id` = `sugarcrm`.`accounts_contacts`.`contact_id`
            ) AS c ON a.CLIENTIDA = c.CLIENTIDC
        ) AS d
    )
INNER JOIN (
    (
        projects AS p
        INNER JOIN orders AS o ON p.ORDERNO = o.ORDERNO
    )
    INNER JOIN category AS cat ON p.category_id = cat.category_id
) ON d.CONTACTID = o.CONTACTID

Explain on this provides the following:

1, PRIMARY, cat, ALL, PRIMARY, , , , 10, 
1, PRIMARY, p, ref, FK_orders_projects,FK_category_projects_idx, FK_category_projects_idx, 5, hdb.cat.category_id, 400, Using where
1, PRIMARY, o, eq_ref, PRIMARY, PRIMARY, 4, hdb.p.ORDERNO, 1, 
1, PRIMARY, <derived2>, ALL, , , , , 18878, Using where
2, DERIVED, <derived3>, ALL, , , , , 7087, 
2, DERIVED, <derived5>, ALL, , , , , 18879, Using where
5, DERIVED, contacts, ALL, , , , , 8261, 
6, UNION, contacts, ALL, , , , , 10251, 
6, UNION, accounts_contacts, ref, idx_contid_del_accid, idx_contid_del_accid, 111, sugarcrm.contacts.id, 1, Using index
, UNION RESULT, <union5,6>, ALL, , , , , , 
3, DERIVED, clients, ALL, , , , , 2296, 
4, UNION, accounts, ALL, , , , , 4548, 
, UNION RESULT, <union3,4>, ALL, , , , , , 

The original query without the union takes 0.125 seconds

SELECT p.PROJID, p.StartDate, o.ORDERNO, p.PROJCODE, p.PROJECT, cat.type AS CATEGORY, p.AREA, p.STATE, p.COUNTRY,
p.VALUE, p.PROCESSOR, p.PROJINFO, p.NES, p.SPECSALE, p.OFFICE, p.DEPTCODE, p.INTERNLCHG, p.INTERCOCHG, p.LORM,
p.PERCENT, a.COMPANY, CONCAT(c.LASTNAME, ", ", c.FIRSTNAME) AS Contact
FROM (clients AS a 
INNER JOIN contacts AS c ON a.CLIENTID =c.CLIENTID) 
INNER JOIN ((projects AS p INNER JOIN orders AS o ON p.ORDERNO = o.ORDERNO) 
INNER JOIN category AS cat ON p.category_id = cat.category_id) ON c.CONTACTID = o.CONTACTID
ORDER BY p.PROJID, a.COMPANY;

explain on this provides following:

1, SIMPLE, cat, ALL, PRIMARY, , , , 10, Using temporary; Using filesort
1, SIMPLE, p, ref, FK_orders_projects,FK_category_projects_idx, FK_category_projects_idx, 5, hdb.cat.category_id, 400, Using where
1, SIMPLE, o, eq_ref, PRIMARY,FK_contacts_orders, PRIMARY, 4, hdb.p.ORDERNO, 1, 
1, SIMPLE, c, eq_ref, PRIMARY,FK_clients_contacts, PRIMARY, 52, hdb.o.CONTACTID, 1, 
1, SIMPLE, a, eq_ref, PRIMARY, PRIMARY, 52, hdb.c.CLIENTID, 1, 

Query with view:

SELECT
    p.PROJID,
    p.StartDate,
    o.ORDERNO,
    p.PROJCODE,
    p.PROJECT,
    cat.type AS CATEGORY,
    p.AREA,
    p.STATE,
    p.COUNTRY,
    p.VALUE,
    p.PROCESSOR,
    p.PROJINFO,
    p.NES,
    p.SPECSALE,
    p.OFFICE,
    p.DEPTCODE,
    p.INTERNLCHG,
    p.INTERCOCHG,
    p.LORM,
    p.PERCENT,
    a.COMPANY,
    CONCAT(
        c.LASTNAME,
        ", ",
        c.FIRSTNAME
    ) AS Contact
FROM
    (
        view_accounts_sugar_hdb AS a
        INNER JOIN view_contacts_sugar_hdb AS c ON a.CLIENTID = c.CLIENTID
    )
INNER JOIN (
    (
        projects AS p
        INNER JOIN orders AS o ON p.ORDERNO = o.ORDERNO
    )
    INNER JOIN category AS cat ON p.category_id = cat.category_id
) ON c.CONTACTID = o.CONTACTID
ORDER BY
    p.PROJID,
    a.COMPANY;

This takes over 340 secs.

Answer 1

This one was definitely uglier than the last one I helped you on :)... Anyhow, the same principles apply. For future, please do try to understand what I'm doing here. Write the JOIN relationships FIRST to know where your data is coming from. Also, take a look at my indentations... I am showing at each level for readability.

Orders -> Projects -> Categories...

then, the path for those in the normal clients table via

Orders -> Contacts -> Clients

finally to the SugarCRM contacts...

Orders -> Accounts_Contacts -> Accounts

So, now that you have the relationships set (and aliased), this follows similar of last answer implementing LEFT-JOIN to normal contact/client vs CRM Contacts/Accounts.

The field list is simple on the Order, Products and Category tables as those are pretty direct. That leaves just the "who/client" information where the LEFT-JOIN comes in. If the Normal client is null, use the CRM version fields, otherwise use the normal client fields...

SELECT
      P.PROJID,
      P.StartDate,
      O.ORDERNO,
      P.PROJCODE,
      P.PROJECT,
      cat.`type` AS CATEGORY,
      P.AREA,
      P.STATE,
      P.COUNTRY,
      P.VALUE,
      P.PROCESSOR,
      P.PROJINFO,
      P.NES,
      P.SPECSALE,
      P.OFFICE,
      P.DEPTCODE,
      P.INTERNLCHG,
      P.INTERCOCHG,
      P.LORM,
      P.PERCENT,
      CASE when HCLIENT.ClientID IS NULL
           then SCLIENT.`name`
           ELSE HCLIENT.Company end as Company,
      CASE when HCLIENT.ClientID IS NULL
           then CONCAT( SCT.LAST_NAME, ", ", SCT.FIRST_NAME ) 
           ELSE CONCAT( HCT.LASTNAME, ", ", HCT.FIRSTNAME ) end as Contact
   FROM
      orders O
         JOIN projects P
            ON O.OrderNo = P.OrderNo
            JOIN category AS cat 
               ON p.category_id = cat.category_id

         LEFT JOIN hdb.contacts HCT
            ON O.ContactID = HCT.ContactID
            LEFT JOIN hdb.clients HCLIENT
               ON HCT.ClientID = HCLIENT.ClientID

         LEFT JOIN sugarcrm.contacts SCT
            ON O.ContactID = SCT.ID
            LEFT JOIN sugarcrm.accounts_contacts SAC
               ON SCT.ID = SAC.contact_id
               LEFT JOIN sugarcrm.accounts SCLIENT
                  ON SCT.account_id = SCLIENT.ID

I'd be interested in the performance improvement too.