Updating 100k records in one update query

Question

Is it possible, or recommended at all, to run one update query, that will update nearly 100k records at once?

If so, how can I do that? I am trying to pass an array to my stored proc, but it seems not to work, this is my SP:

CREATE PROCEDURE [dbo].[UpdateAllClients]
    @ClientIDs varchar(max)
AS
BEGIN
    DECLARE @vSQL varchar(max)
    SET @vSQL = 'UPDATE Clients SET LastUpdate=GETDATE() WHERE ID IN (' + @ClientIDs + ')';
    EXEC(@vSQL);
END

I have not idea whats not working, but its just not updating the relevant queries.

Anyone?

Answer 1

The UPDATE is reading your @ClientIDs ( as a Comma Separated Value ) as a whole. To illustrate it more, you are doing like this.

assume the @ClientIDs = 1,2,3,4,5

your UPDATE command is interpreting it like this

UPDATE Clients SET LastUpdate=GETDATE() WHERE ID IN ('1,2,3,4,5')';

and not

UPDATE Clients SET LastUpdate=GETDATE() WHERE ID IN (1,2,3,4,5)';

One suggestion to your question is by using subquery on your UPDATE , example

UPDATE Clients 
   SET LastUpdate = GETDATE() 
WHERE ID IN
    (
       SELECT ID
       FROM tableName
       -- where condtion
    )

Hope this makes sense.

Answer 2

A few notes to be aware of.

Big updates like this can lock up the target table. If > 5000 rows are affected by the operation, the individual row locks will be promoted to a table lock, which would block other processes. Worth bearing in mind if this could cause an issue in your scenario. See: Lock Escalation

With a large number of rows to update like this, an approach I'd consider is (basic):

1. bulk insert the 100K Ids into a staging table (eg from .NET, use SqlBulkCopy)
2. update the target table, using a join onto the above staging table
3. drop the staging table

This gives some more room for controlling the process, but breaking the workload up into chunks and doing it x rows at a time.

Answer 3

There is a limit for the number of items you pass to 'IN' if you are giving an array.

So, if you just want to update the whole table, skip the IN condition.

If not specify an SQL inside IN. That should do the job

Answer 4

The database will very likely reject that SQL statement because it is too long.

When you need to update so many records at once, then maybe your database schema isn't appropriate. Maybe the LastUpdate datum should not be stored separately for each client but only once globally or only once for a constant group of clients?

But it's hard to recommend a good course of action without seeing the whole picture.

Answer 5

What version of sql server are you using? If it is 2005+ I would recommend using TVPs (table valued parameters - http://msdn.microsoft.com/en-us/library/bb510489.aspx ). The transfer of data will be faster (as opposed to building a huge string) and your query would look nicer:

update c
set lastupdate=getdate()
from clients c
join @mytvp t on c.Id = t.Id

Answer 6

Each SQL statement on its own is a transaction statement . This means sql server is going to grab locks for all these million of rows .It can really degrade the performance of a table .So you really don't tend to update a table which has million of rows in it which hurts the performance.So the workaround is to set rowcount before DML operation

set rowcount=100
UPDATE Clients SET LastUpdate=GETDATE()
WHERE ID IN ('1,2,3,4,5')';
set rowcount=0

or from SQL server 2008 you can parametrize Top keyword

Declare @value int
set @value=100000 
again:
UPDATE top (@value) Clients SET LastUpdate=GETDATE()
WHERE ID IN ('1,2,3,4,5')';
if @@rowcount!=0  goto again 

See how fast the above query takes and then adjust and change the value of the variable .You need to break the tasks for smaller units as suggested in the above answers

Answer 7

Method 1:

1. Split the @clientids with delimiters ','
2. put in array and iterate over that array
3. update clients table for each id.

OR

Method 2:

Instead of taking @clientids as a varchar2, follow below steps

1. create object type table for ids and use join.
2. For faster processing u can also create index on clientid as well.