Is it possible to use cross database queries in both Azure and SQL?

Question

I am working on a project designed to run on azure platform using Azure database and Locally using local sql server. We have stored procedure that contain cross database calls. However It doesn't works on Azure server.I need cross database queries that capable of working in both azure and local sql server.

We an use Elastic Query which allows us to query across Azure SQL Databases

https://azure.microsoft.com/en-us/documentation/articles/sql-database-elastic-query-overview/

We can setup external data source in azure server Using following code. In this case we can execute cross database calls likes joining tables in a single database.

 CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'yourPassword';  

 CREATE DATABASE SCOPED CREDENTIAL yourServeradminlogin  
 WITH IDENTITY = 'yourServeradminlogin',  
 SECRET = 'yourPassword';  

 CREATE EXTERNAL DATA SOURCE RefmyDemoDB2  
 WITH  
 (  
    TYPE=RDBMS,  
    LOCATION='testdbdemoserver.database.windows.net',  
    DATABASE_NAME='myDemoDB2',  
    CREDENTIAL= yourServeradminlogin  
 );  

 CREATE EXTERNAL TABLE [dbo].[Department](  
    [DeptId] [int] NOT NULL,  
    [Name] [varchar](50) NULL  
 )  
 WITH  
 (  
    DATA_SOURCE = RefmyDemoDB2  
 );  

It doesn't seem possible to use above method on our local SQL server without adding a third party database engine. Install something called PolyBase to do cross database query in local sql server version 2017 and above.It only supported with external data source of type such as HADOOP rather than referencing another database(or any data storage) within the sql server.

We can accomplish that using following code

  CREATE MASTER KEY ENCRYPTION BY PASSWORD='MyP@ssword123secretword';

  CREATE DATABASE SCOPED CREDENTIAL mycredential  
  WITH IDENTITY = 'credential', Secret = 'secretkey' 

  CREATE EXTERNAL DATA SOURCE mycustomers
  WITH (
    TYPE = HADOOP,
    LOCATION =  'wasbs://azurestorage.blob.core.windows.net/',
    CREDENTIAL = mycredential
  );

  CREATE EXTERNAL FILE FORMAT csvformat 
  WITH ( 
    FORMAT_TYPE = DELIMITEDTEXT, 
    FORMAT_OPTIONS ( 
      FIELD_TERMINATOR = ','
    ) 
  );

  CREATE EXTERNAL TABLE TableName
  ( 
    [did] [int] NOT NULL,  
    [Dname] [varchar] (50) NULL 
  ) 
  WITH 
  ( 
     LOCATION = '/', 
     DATA_SOURCE = mycustomers, 
     FILE_FORMAT = csvformat
  )

Using HADOOP and polybase we can create an external data source in local sql server. But it create external data source of that external data storage. That is External data table located in that external storage.Exactly, my requirement is create an external data source of database within the sql server. So that i can use same corss data base query in both azure and local sql server

Is there any solution to solve this. or any solution to run cross database queries in both azure and local sql server?

Answer 1

You should not assume that using external tables in SQL Azure is remotely similar to doing cross-database queries in SQL Server. They are not the same thing and they have very different performance profiles. External tables is closer to linked servers than cross-database queries in SQL Server.

In SQL Server, cross-database queries are: - running in the same SQL instance - run the same basic execution code path to read data vs. single-database SQL Azure - you have some slightly different transactional semantics vs. single-database operaitons in SQL Server, but the experience is generally similar from your perspective.

In SQL Azure singletons ("traditional") SQL Azure, you generally do not have the databases on the same physical machines. So, you have to cross-server queries (which are exposed using a linked-server like mechanism called external tables which support sharding/fan-out scenarios mostly). Trying to use this feature to simulate cross-database queries may functionally work, but it is not really a great plan due to the performance differences. There are also no real transactional guarantees in this path at all (no DTC).

SQL Azure Managed Instance does support cross-database queries within a single SQL Server instance internally. So, this would be the path that would be most similar for you to use if you really want to use cross-database queries. If you want to use SQL Azure single databases, you generally would not want to do cross-db queries at all for a legacy workload and you would want to rewrite to avoid the dependency. (Otherwise it will just be an ongoing headache due to the performance differences)