SQL - improve NOT EXISTS query performance

Question

Is there a way I can improve this kind of SQL query performance:

INSERT
INTO ...
WHERE NOT EXISTS(Validation...)

The problem is when I have ma

Answer 1

Outer Apply tends to work for me...

instead of:

from t1
where not exists (select 1 from t2 where t1.something=t2.something)

I'll use:

from t1
outer apply (
    select top 1 1 as found from t2 where t1.something=t2.something
) t2f
where t2f.found is null

Answer 2

insert into customers 
select * 
from newcustomers 
where customerid not in (select customerid 
                         from customers)

..may be more efficient. As others have said, make sure you've got indexes on any lookup fields.

Answer 3

Make sure you are searching on indexed columns, with no manipulation of the data within those columns (like substring etc.)

Answer 4

If you can at all reduce your problem space, then you'll gain heaps of performance. Are you absolutely sure that every one of those rows in that table needs to be checked?

The other thing you might want to try is a DELETE InsertTable FROM InsertTable INNER JOIN ExistingTable ON <Validation criteria> before your insert. However, your mileage may vary

Answer 5

Try to replace the NOT EXISTS with a left outer join, it sometimes performs better in large data sets.

Answer 6

Off the top of my head, you could try something like:

 TRUNCATE temptable
 INSERT INTO temptable ...
 INSERT INTO temptable ... 
 ...
 INSERT INTO realtable
 SELECT temptable.* FROM temptable
 LEFT JOIN realtable on realtable.key = temptable.key
 WHERE realtable.key is null