Why the Select * FROM Table where ID NOT IN ( list of int ids) query is slow in sql server ce?

Question

well this problem is general in sql server ce
i have indexes on all the the fields.
also the same query but with ID IN ( list of int ids) is pretty fast.
i tried to change the query to OUTER Join but this just make it worse. so any hints on why this happen and how to fix this problem?

Answer 1

That's because the index is not really helpful for that kind of query, so the database has to do a full table scan. If the query is (for some reason) slower than a simple "SELECT * FROM TABLE", do that instead and filter the unwanted IDs in the program.

EDIT: by your comment, I recognize you use a subquery instead of a list. Because of that, there are three possible ways to do the same (hopefully one of them is faster):

Original statement:

select * from mytable where id not in (select id from othertable);

Alternative 1:

select * from mytable where not exists 
   (select 1 from othertable where mytable.id=othertable.id);

Alternative 2:

select * from mytable
minus
select mytable.* from mytable in join othertable on mytable.id=othertable.id;

Alternative 3: (ugly and hard to understand, but if everything else fails...)

select * from mytable
  left outer join othertable on (mytable.id=othertable.id)
  where othertable.id is null;

Answer 2

This is not a problem in SQL Server CE, but overall database.

The OPERATION IN is sargable and NOT IN is nonsargable.

What this mean ?

Search ARGument Able, thies mean that DBMS engine can take advantage of using index, for Non Search ARGument Ablee the index can't be used.

The solution might be using filter statement to remove those IDs

More in SQL Performance Tuning by Peter Gulutzan.

Answer 3

ammoQ is right, index does not help much with your query. Depending on distribution of values in your ID column you could optimise the query by specifying which IDs to select rather than not to select. If you end up requesting say more than ~25% of the table index will not be used anyway though because for nonclustered indexed (which is the only type of indexes which SQL CE supports if memory serves) it would be cheaper to scan the table. Otherwise (if the query is actually selective) you could re-write query with ID ranges to select ('union all' may work better than 'or' to combine ranges if SQL CE supports 'union all', not sure)