MySQL Join Very Slow

Question

We have a problem with one of our MySQL statements.

Basically the following statement is taking 5 seconds to run. We have diagnosed it is down to the join of two select statement. When the select statements are run individually they take only 0.2 seconds but when combined with the JOIN it takes 5 seconds.

Is there anything you can see that we are doing wrong or can you see a better way?

Indexes have been added to all columns contained in the join but it does not effect the speed

SELECT temp_4.primaryid, temp_1.`subjectID` , temp_4.`testOccasionID` 
,`studyNumbers` ,`testDate` 

FROM (
   SELECT * FROM (
     SELECT primarys.primaryid , q_1 AS `subjectID` , q_2 AS `studyNumbers` FROM 
primarys LEFT OUTER JOIN questions_1_100 ON primarys.primaryid = 
questions_1_100.primaryid WHERE 0 = 0 AND q_1 IS NOT NULL GROUP BY primaryid) AS 
maintable_1 
GROUP BY `subjectID` ) AS temp_1 

JOIN 

(SELECT * FROM 
(SELECT primarys.primaryid , q_1 AS `subjectID` , q_4 AS `testOccasionID` , 
DATE_FORMAT(q_5, '%m/%d/%Y') AS `testDate` FROM primarys LEFT OUTER JOIN 
questions_1_100 ON primarys.primaryid = questions_1_100.primaryid WHERE 0 = 0 AND 
q_1 IS NOT NULL AND q_4 IS NOT NULL GROUP BY primaryid) AS maintable_4 
GROUP BY `subjectID` ,`testOccasionID` ) AS temp_4 

ON temp_1.`subjectID` = temp_4.`subjectID` 

Table definitions:

CREATE TABLE primarys 
( primaryid BIGINT(20) NOT NULL AUTO_INCREMENT,
  dateinserted DATETIME,
  datemodified DATETIME,
  useridinserted BIGINT(20),
  useridmodified BIGINT(20),
  locked VARCHAR(1) NOT NULL DEFAULT 0, 
  primaryquestionlinks TEXT, 
  PRIMARY KEY (primaryid), 
  FOREIGN KEY (useridinserted) REFERENCES users (userid) ON UPDATE CASCADE ON DELETE SET NULL, 
  FOREIGN KEY (useridmodified) REFERENCES users (userid) ON UPDATE CASCADE ON DELETE SET NULL ) ENGINE=InnoDB;

CREATE TABLE questions_1_100
( primaryid BIGINT(20) NOT NULL,
  q_1 BIGINT(20),
  q_2 VARCHAR(50),
  q_3 BIGINT(20),
  q_4 BIGINT(20),
  q_5 DATE,
  PRIMARY KEY (primaryid),
  FOREIGN KEY (primaryid) REFERENCES primarys (primaryid) ON UPDATE CASCADE ON DELETE CASCADE ) ENGINE=InnoDB;

Data for a single marching subject is as follows - the order is primaryid,q_1,q_2,q_4,q_5:

1    1    01001    NULL    NULL
7286 1    NULL     1       1997-12-18
7287 1    NULL     2       1998-02-25

The required output is:

7286 1    01001    1      1997-12-18
7287 1    01001    2      1998-02-25

Many thanks

to take this a step further... what if q_1 and q_4 were in two separate tables. such as the following table structure. The only way i can think of is to add left outer joins and several sub-queries?

CREATE TABLE primarys 
( primaryid BIGINT(20) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (primaryid));

CREATE TABLE questions_1_100
( primaryid BIGINT(20) NOT NULL,
q_1 BIGINT(20),
q_2 VARCHAR(50),
PRIMARY KEY (primaryid));

CREATE TABLE questions_101_200
( primaryid BIGINT(20) NOT NULL,
q_4 BIGINT(20),
q_5 DATE,
PRIMARY KEY (primaryid));

INSERT INTO primarys values (1);
INSERT INTO primarys values (7286);
INSERT INTO primarys values (7287);

INSERT INTO questions_1_100 VALUES (1,'1','01001');
INSERT INTO questions_1_100 VALUES (7286,'1','');
INSERT INTO questions_1_100 VALUES (7287,'1','');

INSERT INTO questions_101_200 VALUES (7286,'1','1997-12-18');
INSERT INTO questions_101_200 VALUES (7287,'2','1998-02-25');

Answer 1

Try this query

You are creating second inner query which is of no use. it would be better to remove that one.

Create these indexes table questions_1_100 compund index (primaryid, q_1, q_4)

table primarys index (primaryid)

Hope this helps..

SELECT 
  temp_4.primaryid, 
  temp_1.`subjectID` , 
  temp_4.`testOccasionID` ,
  `studyNumbers` ,
  `testDate` 
FROM (
  SELECT 
    primarys.primaryid , 
    q_1 AS `subjectID` , 
    q_2 AS `studyNumbers` 
  FROM 
    primarys 
  LEFT OUTER JOIN 
    questions_1_100 
  ON 
    primarys.primaryid = questions_1_100.primaryid 
  WHERE 
    q_1 IS NOT NULL 
  GROUP BY 
    primaryid, 
    subjectID) AS temp_1 
JOIN 
  (SELECT  
     primarys.primaryid , 
     q_1 AS `subjectID` , 
     q_4 AS `testOccasionID` , 
     DATE_FORMAT(q_5, '%m/%d/%Y') AS `testDate` 
  FROM 
     primarys 
  LEFT OUTER JOIN 
     questions_1_100 
  ON 
     primarys.primaryid = questions_1_100.primaryid 
  WHERE 
     q_1 IS NOT NULL AND 
     q_4 IS NOT NULL 
  GROUP BY 
     primaryid,
     subjectID,
     testOccasionID) AS temp_4 
ON 
  temp_1.`subjectID` = temp_4.`subjectID` 

Answer 2

Try:

SELECT p.primaryid , 
       q2.q_1 AS `subjectID`, 
       q4.q_4 AS `testOccasionID`, 
       q2.q_2 AS `studyNumbers`,
       DATE_FORMAT(q4.q_5, '%m/%d/%Y') AS `testDate` 
FROM primarys p
JOIN questions_1_100 q2
  ON p.primaryid = q2.primaryid
JOIN questions_1_100 q4 
  ON q2.q_1 = q4.q_1 and q4.q_4 is not null

SQLFiddle here .

A simpler version, with the test occasion primaryid:

SELECT q4.primaryid , 
       q2.q_1 AS `subjectID`, 
       q4.q_4 AS `testOccasionID`, 
       q2.q_2 AS `studyNumbers`,
       DATE_FORMAT(q4.q_5, '%m/%d/%Y') AS `testDate` 
FROM questions_1_100 q2
JOIN questions_1_100 q4 
  ON q2.q_1 = q4.q_1 and q4.q_4 is not null
where q2.q_2 is not null

SQLFiddle here .