Azure Data Factory Copy Identity Column With Gaps

Question

I created a pipeline and two linked services to move data from an on-prem instance of SQL Server to an Azure Sql instance. The issue I\'m running into is that we have a table \

Answer 1

Azure Data Factory does not natively support switching the identity property of tables on or off, but two workarounds spring to mind.

1. Use Data Factory to load the data into a staging table (where identity property is not set) then use a Stored Proc task to call a stored procedure where you have much tighter control, including the ability to set the identity property on or off.
2. If you are using Azure SQL Database (or SQL Server on a VM), you could use table-valued parameters and pass your data into the stored proc task that way, skipping the staging table. This technique does not work with Azure SQL Data Warehouse. I probably would not recommend this for high volume. This example shows how:

https://github.com/Microsoft/azure-docs/blob/master/includes/data-factory-sql-invoke-stored-procedure.md

I have not been able to test these but believe they would work. Let me know if you have any issues.

Answer 2

I accepted wBob's answer but wanted to put a little more detail into what I did.

I had probably 100 tables to move over with all sorts of dependencies and identities. So here are the steps I carried out to get the data into azure:

1. Create a pipeline to move over all tables with no identity and no dependencies, found by querying sys.tables:

   select *
   from sys.tables t
   where not exists (
       select *
       from sys.columns c 
       where c.object_id = t.object_id
       and is_identity = 1
   )
   

   and bumped the results here up against the results of sp_msdependencies where the oType = 8. I then took all of the tables in this result set where oSequence = 1 (no dependencies) and put those tables in the pipeline and ran it.

2. I created a Staging schema and re-created all of the tables with an identity column (found by removing the 'not' in the query in (1.), and there were over 60 of them) and removed the identity specification when creating them.

3. I then created another data factory pipeline to move the data into these Staging tables.

4. Ran a bunch of 'insert into...' statements to move the data from the staging tables into their identity-laden counterparts, setting identity_insert on and off each time. NOTE: Here, I also had to be mindful of the sp_msdependencies result so as not to get foreign errors

5. Created a data factory pipeline to move the remaining tables over.

Whew...