C# Efficiently delete 50000 records in batches using SQLBulkCopy or equivalent library

Question

I\'m using this library to perform bulk delete in batches like following:

  while (castedEndedItems.Any())
  {
    var subList = castedEndedItems.Take(4000).         


        

Answer 1

any EF solution here is probably going to perform lots of discreet operations. Instead, I would suggest manually building your SQL in a loop, something like:

using(var cmd = db.CreateCommand())
{
    int index = 0;
    var sql = new StringBuilder("delete from [SomeTable] where [SomeId] in (");
    foreach(var item in items)
    {
        if (index != 0) sql.Append(',');
        var name = "@id_" + index++;
        sql.Append(name);
        cmd.Parameters.AddWithValue(name, item.SomeId);            
    }
    cmd.CommandText = sql.Append(");").ToString();
    cmd.ExecuteNonQuery();
}

You may need to loop this in batches, though, as there is an upper limit on the number of parameters allowed on a command.

Answer 2

If you are prepared to use a stored procedure then you can do this without any external library:

• Create the sproc using a table valued parameter @ids
• Define a SQL type for that table valued parameter (just an id column assuming a simple PK)

• In the sproc use

  delete from table where id in (select id from @ids);
  

• In your application create a DataTable and populate to match the SQL table

• Pass the data table as an command parameter when calling the sproc.

This answer illustrates the process.

Any other option will need to do the equivalent of this – or something less efficient.

Answer 3

If you don't mind the extra dependency, you could use the NuGet package Z.EntityFramework.Plus.

The code is roughly as follows:

using Z.EntityFramework.Plus;
[...]
         using (yourDbContext context = new yourDbContext())
         {
              yourDbContext.yourDbSet.Where( yourWhereExpression ).Delete();
         }

It is simple and efficient. The documentation contains exact numbers about the performance.

Regarding licensing: As far as I know, version 1.8 has an MIT license: https://github.com/zzzprojects/EntityFramework-Plus/blob/master/LICENSE The newer version are not free to use.