How can I make this update query more efficient

Question

I'm using SQLite3. Can anyone see how to make this query more efficient? Essentially, I have a table into which I insert some records with a given InsertDate. The table might already contain duplicates (on 4 other fields) to the newly inserted records. I want to update a certain field in all those duplicates

To find the duplicate rows I use this query (identifiers changed for clarity)

SELECT Table1.rowid
FROM   Table1 
INNER JOIN
      (
       SELECT Field1, Field2, Field3, Field4, InsertDate
       FROM Table1
       WHERE InsertDate = '2013-07-11' 
       ) AS T
ON
       T.Field1 = Table1.Field1
   AND T.Field2 = Table1.Field2
   AND T.Field3 = Table1.Field3
   AND T.Field4 = Table1.Field4

WHERE Table1.InsertDate <> T.InsertDate  --(could use WHERE Table1.InsertDate <> '2013-07-11')

That select query takes 6s to run with indexes on Fields 1,2 and 3 (interestingly I found having an index on Field4 increased the time to 9s so I didn't have one)

I then use that query in the update query, deciding which records to update by having the select following an IN clause in the update statement ie

UPDATE Table 1
SET Field 5 = 'A'
WHERE Table1.RowID IN

    -- here comes the original select query
    (
    SELECT Table1.rowid
    FROM   Table1 
    INNER JOIN
       (
       SELECT FIELD1, Field2, Field3, Field4, InsertDate
       FROM Table1
       WHERE InsertDate = '2013-07-11' 
       ) AS T
    ON
        T.Field1 = Table1.Field1
    AND T.Field2 = Table1.Field2
    AND T.Field3 = Table1.Field3
    AND T.Field4 = Table1.Field4

    WHERE Table1.InsertDate <> T.InsertDate
    )

But this take nearly 25 seconds to run!

I tried making a temp table out of the intial SELECT, intending to use that in the update query but that also took around 25s to make the table so it doen't look like its the actual updating that's slowing things down, it seens to be using my 6s select query with 'IN'.

Typically I would be updating about 9000 rows out of about 300,000.

Any suggestions to improve this query would be very welcome.

Answer 1

If I understand the logic correctly, you don't need rowid :

UPDATE Table 1
    SET Field 5 = 'A'
    WHERE InsertDate <> '2013-07-11' and
          EXISTS (select 1
                  from table1 tt
                  where tt.InsertDate = '2013-07-11' and
                        T.Field1 = tt.Field1 and
                        T.Field2 = tt.Field2 and
                        T.Field3 = tt.Field3 and
                        T.Field4 = tt.Field4        
                 );

Then, create an index on table1(field1, field2, field3, field4, insertdate) and table1(insertdate) .