Using “IN” in a WHERE clause where the number of items in the set is very large

Question

I have a situation where I need to do an update on a very large set of rows that I can only identify by their ID (since the target records are selected by the user and have noth

Answer 1

Another alternative is to store those numbers in a temp table and use it in a join to do the update. If you are able to execute a single update statement is definitely better than executing one statement per record.

Answer 2

Without knowing what a "very large" number of ID's might be, I'd venture a guess. ;-)

Since you are using Access as a database, the number of ID's can't be that high. Assuming we're talking about less than, say 10,000 numbers and we should know the limitations of the containers to hold the ID's (what language is used for the front end?), I'd stick to one UPDATE statement; if that is most readable and easiest to perform maintenance on later. Otherwise I'd split them into multiple statements using some clever logic. Something like split the statement into multiple statements with in one, ten, hundred, thousand... ID's per statement.

Then, I'd leave it to the DB optimiser to execute the statement(s) as efficient as possible. I would probably do an 'explain' on the query / queries to make sure nothing silly is going on though.

But in my experience, it is quite often OK to leave this kind of optimisation to the database manager itself. The one thing that takes the most time is usually the actual connection to the database, so if you can execute all queries within the same connection it is normally no problems. Make sure you send off all UPDATE statements before you start to look into and wait for any result sets coming back though. :-)

Answer 3

There are multiple ways of accommodating a large set of values in a where condition

1. Using Temp Tables

   Insert the values into a temp table with a single column.

   Create a UNIQUE INDEX on that particular column.

   INNER JOIN the required table with the newly created temp table

2. Using array-like functionality in SQL Server

   SQL does support an array like functionality

   check this link for full documentation.

SAMPLE SYNTAX :

Create TABLE #IDs (id int NOT NULL)
DECLARE @x varchar(max) = '' 
DECLARE @xParam XML;
SELECT @xParam = CAST('<i>' + REPLACE(@x, ',', '</i><i>') + '</i>' AS XML)
INSERT into #IDs
SELECT x.i.value('.','NVARCHAR(100)') as key FROM @xParam .nodes('//i') x(i)
CREATE UNIQUE INDEX IX_#IDs ON #IDs (ID ASC) 

Query using

SELECT A.Name, A.Age from Table A 

INNER JOIN #IDs id on id.id = A.Key

Answer 4

I would always use

WHERE id IN (1,2,3,4,.....10000)

unless your in clause was stupidly large, which shouldn't really happen from user input.

edit: For instance, Rails does this a lot behind the scenes

It would definitely not be better to do separate update statements in a single transaction.