Speed up LINQ inserts

Question

I have a CSV file and I have to insert it into a SQL Server database. Is there a way to speed up the LINQ inserts?

I\'ve created a simple Repository method to save a rec

Answer 1

Have a look at the following page for a simple walk-through of how to change your code to use a Bulk Insert instead of using LINQ's InsertOnSubmit() function.

You just need to add the (provided) BulkInsert class to your code, make a few subtle changes to your code, and you'll see a huge improvement in performance.

Mikes Knowledge Base - BulkInserts with LINQ

Good luck !

Answer 2

Have you tried wrapping the inserts within a transaction and/or delaying db.SubmitChanges so that you can batch several inserts?

Transactions help throughput by reducing the needs for fsync()'s, and delaying db.SubmitChanges will reduce the number of .NET<->db roundtrips.

Edit: see http://www.sidarok.com/web/blog/content/2008/05/02/10-tips-to-improve-your-linq-to-sql-application-performance.html for some more optimization principles.

Answer 3

Why not pass an offer[] into that method, and doing all the changes in cache before submitting them to the database. Or you could use groups for submission, so you don't run out of cache. The main thing would be how long till you send over the data, the biggest time wasting is in the closing and opening of the connection.

Answer 4

I suspect it isn't the inserting or updating operations that are taking a long time, rather the code that determines if your offer already exists:

Offer dbOffer = this.db.Offers.SingleOrDefault (
         o => o.offer_id == offer.offer_id);

If you look to optimise this, I think you'll be on the right track. Perhaps use the Stopwatch class to do some timing that will help to prove me right or wrong.

Usually, when not using Linq-to-Sql, you would have an insert/update procedure or sql script that would determine whether the record you pass already exists. You're doing this expensive operation in Linq, which certainly will never hope to match the speed of native sql (which is what's happening when you use a SqlCommand and select if the record exists) looking-up on a primary key.

Answer 5

Converting this to a compiled query is the easiest way I can think of to boost your performance here:

Change the following:

    Offer dbOffer = this.db.Offers.SingleOrDefault (
         o => o.offer_id == offer.offer_id);

to:

Offer dbOffer = RetrieveOffer(offer.offer_id);

private static readonly Func<DataContext, int> RetrieveOffer
{
   CompiledQuery.Compile((DataContext context, int offerId) => context.Offers.SingleOrDefault(o => o.offer_id == offerid))
}

This change alone will not make it as fast as your ado.net version, but it will be a significant improvement because without the compiled query you are dynamically building the expression tree every time you run this method.

As one poster already mentioned, you must refactor your code so that submit changes is called only once if you want optimal performance.

Answer 6

SubmitChanges does not batch changes, it does a single insert statement per object. If you want to do fast inserts, I think you need to stop using LINQ.

While SubmitChanges is executing, fire up SQL Profiler and watch the SQL being executed.

See question "Can LINQ to SQL perform batch updates and deletes? Or does it always do one row update at a time?" here: http://www.hookedonlinq.com/LINQToSQLFAQ.ashx

It links to this article: http://www.aneyfamily.com/terryandann/post/2008/04/Batch-Updates-and-Deletes-with-LINQ-to-SQL.aspx that uses extension methods to fix linq's inability to batch inserts and updates etc.