C# and SQLServer normalizing large sets of Urls

Question

I have many tables in the database that have at least one column that contains a Url. And these are repeated a lot through-out the database. So I normalize them to a dedicated t

Answer 1

There is even a better way than SQLBulkCopy.

It's called Structured Parameters and it allows you to pass a table-valued parameter to stored procedure or query through ADO.NET.

There are code examples in the article, so I will only highlight what you need to do to get it up and working:

1. Create a user defined table type in the database. You can call it UrlTable
2. Setup a SP or query which does the SELECT by joining with a table variable or type UrlTable
3. In your backing code (C#), create a DataTable with the same structure as UrlTable, populate it with URLs and pass it to an SqlCommand through as a structured parameter. Note that column order correspondence is critical between the data table and the table type.

What ADO.NET does behind the scenes (if you profile the query you can see this) is that before the query it declares a variable of type UrlTable and populates it (INSERT statements) with what you pass in the structured parameter.

Other than that, query-wise, you can do pretty much everything with table-valued parameters in SQL (join, select, etc).

Answer 2

I think you could use the IGNORE_DUP_KEY option on your index. If you set IGNORE_DUP_KEY = ON on the index of the URL column, the duplicate values are simply ignored and the rest are inserted appropriately.