Can I do a very large insert with Linq-to-SQL?

Question

I\'ve got some text data that I\'m loading into a SQL Server 2005 database using Linq-to-SQL using this method (psuedo-code):

Create a DataContext

While (ne         


        

Answer 1

It looks like this would work however the changes (and thus memory) that are kept by the DataContext are going to grow with each InsertOnSubmit. Maybe it's adviseable to perform a SubmitChanges every 100 records?

I would also take a look at SqlBulkCopy to see if it doesn't fit your usecase better.

Answer 2

IF you need to do bulk inserts, you should check out SqlBulkCopy

Linq-to-SQL is not really suited for doing large-scale bulk inserts.

Answer 3

Just for the record I did as marc_s and Peter suggested and chunked the data. It's not especially fast (it took about an hour and a half as Debug configuration, with the debugger attached and quite a lot of console progress output), but it's perfectly adequate for our needs:

Create a DataContext

numRows = 0;
While (new data exists)
{
    Read a record from the text file

    Create a new Record 

    Populate the record

    dataContext.InsertOnSubmit(record)

    // Submit the changes in thousand row batches
    if (numRows % 1000 == 999)
        dataContext.SubmitChanges()

    numRows++
}

dataContext.SubmitChanges()

Answer 4

You would want to call SubmitChanges() every 1000 records or so to flush the changes so far otherwise you'll run out of memory.

If you want performance, you might want to bypass Linq-To-SQL and go for System.Data.SqlClient.SqlBulkCopy instead.