Proper Indexing MySQL Table

Question

I can't seem to get this query to perform any faster than 8 hours ! 0_0 I have read up on indexing and I am still not sure I am doing this right.

I am expecting my query to calculate a value for BROK_1_RATING based on dates and other row values - 500,000 records .

Using record #1 as an example - my query should:

• get all other records that have the same ESTIMID

• ignore records where ANALYST =""

• ignore records where ID is the same as record being compared ie ID != 1

• the records must fall within a time frame ie BB. ANNDATS_CONVERTED <= working . ANNDATS_CONVERTED , BB. REVDATS_CONVERTED > working . ANNDATS_CONVERTED

• BB. IRECCD must = 1

• Then count the result

• Then write the count value to the BROK_1_RATING column for record #1

• now do same for record#2, and #3 and so on for the entire table

In human terms - "Examine the date of record #1 - Now, within time frame from record #1 - count the number of times the number 1 exists with the same brokerage ESTIMID , do not count record #1, do not count blank ANALYST rows. Move on to record #2 and do the same"

---

UPDATE `working` SET `BROK_1_RATING` = 

    (SELECT COUNT(`ID`) FROM (SELECT `ID`, `IRECCD`, `ANALYST`,  `ESTIMID`, `ANNDATS_CONVERTED`, `REVDATS_CONVERTED` FROM `working`) AS BB 

        WHERE 
            BB.`ANNDATS_CONVERTED` <= `working`.`ANNDATS_CONVERTED` 
        AND 
            BB.`REVDATS_CONVERTED` > `working`.`ANNDATS_CONVERTED`
        AND 
            BB.`ID` != `working`.`ID`
        AND 
            BB.`ESTIMID` = `working`.`ESTIMID`
        AND
            BB.`ANALYST` != ''
        AND
            BB.`IRECCD` = 1
    )

WHERE `working`.`ANALYST` != '';

---

| ID | ANALYST |   ESTIMID    | IRECCD | ANNDATS_CONVERTED | REVDATS_CONVERTED |  BROK_1_RATING  | NO_TOP_RATING |  
------------------------------------------------------------------------------------------------------------------
| 1  |  DAVE   | Brokerage000 |   4    |    1998-07-01     |    1998-07-04     |                 |      3        |
| 2  |  DAVE   | Brokerage000 |   1    |    1998-06-28     |    1998-07-10     |                 |      4        |
| 3  |  DAVE   | Brokerage000 |   5    |    1998-07-02     |    1998-07-08     |                 |      2        |
| 4  |  DAVE   | Brokerage000 |   1    |    1998-07-04     |    1998-12-04     |                 |      3        |
| 5  |  SAM    | Brokerage000 |   1    |    1998-06-14     |    1998-06-30     |                 |      4        |
| 6  |  SAM    | Brokerage000 |   1    |    1998-06-28     |    1999-08-08     |                 |      4        |
| 7  |         | Brokerage000 |   1    |    1998-06-28     |    1999-08-08     |                 |      5        |
| 8  |  DAVE   | Brokerage111 |   2    |    1998-06-28     |    1999-08-08     |                 |      3        |

'EXPLAIN' results:

id| select_type        | table            | type  | possible_keys | key                   | key_len | ref   | rows   | Extra
----------------------------------------------------------------------------------------------------------------------------------------
1 | PRIMARY            | working          | index | ANALYST       | PRIMARY               | 4       | NULL  | 467847 | Using where
2 | DEPENDENT SUBQUERY | <derived3>       | ALL   | NULL          | NULL                  | NULL    | NULL  | 467847 | Using where
3 | DERIVED            | working          | index | NULL          | test_combined_indexes | 226     | NULL  | 467847 | Using index

I have indexes on the single columns - and as well - have tried multiple column index like this:

ALTER TABLE `working` ADD INDEX `test_combined_indexes` (`IRECCD`, `ID`, `ANALYST`, `ESTIMID`, `ANNDATS_CONVERTED`, `REVDATS_CONVERTED`) COMMENT '';

Answer 1

Well you can shorten the query a lot by just removing the extra stuff:

UPDATE `working` as AA SET `BROK_1_RATING` = 
(SELECT COUNT(`ID`) FROM `working` AS BB 
    WHERE BB.`ANNDATS_CONVERTED` <= AA.`ANNDATS_CONVERTED` 
    AND BB.`REVDATS_CONVERTED` > AA.`ANNDATS_CONVERTED`
    AND BB.`ID` != AA.`ID`
    AND BB.`ESTIMID` = AA.`ESTIMID`
    AND BB.`ANALYST` != ''
    AND BB.`IRECCD` = 1 )
WHERE `ANALYST` != '';