How to Map JSON data from a REST API to Azure SQL using Data Factory

问题

I have a new pipeline in azure data factory. I created the dataset, one from the rest api (a public one): https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=MSFT&apikey=demo

and then I created an azure sql table with columns shown in the screenshot

The problem, is that I dont know how to do the mapping, as this is a complex JSON object, I am limited with the Mapping Designer:

How do I map the date?

回答1:

I tend to use an ELT approach for these, calling the REST API with a Web task and storing the JSON in a SQL table and then shredding the JSON using SQL functions like OPENJSON.

Example pipeline:

The key to getting this approach to work is the expression on the stored procedure parameter. This takes the whole JSON output from the Web task and passes it in to the proc. This is a simple logging proc which inserts the record into a logging table:

@string(activity('Web1').output)

I log to a table and then shred the JSON or you could use OPENJSON directly on the stored proc parameter, eg

--INSERT INTO ...
SELECT
    CAST( [key] AS DATE ) AS timeSeriesDate,
    JSON_VALUE ( x.[value], '$."1. open"' ) AS [open],
    JSON_VALUE ( x.[value], '$."2. high"' ) AS [high],
    JSON_VALUE ( x.[value], '$."3. low"' ) AS [low],
    JSON_VALUE ( x.[value], '$."4. close"' ) AS [close],
    JSON_VALUE ( x.[value], '$."5. volume"' ) AS [volume]

FROM dbo.myLog
    CROSS APPLY OPENJSON(logDetails , '$."Time Series (Daily)"' ) x
--WHERE logId = 23333;

My results:

回答2:

Does the data have a structure? If so, you can generate a dummy file, place it in sink and do a one time mapping. If not, you can Lookup on the file, iterate over the content in a For Each Loop Container and insert details on to a SQL table.

E.g.

insert <<your table>> 
select '@item().name', '@item().address.city', @item().value

The important thing to remember is to iterate at the correct array. Let me know if it's not clear. Not in front of a system right now, so can't add screenshots.

来源：https://stackoverflow.com/questions/60300293/how-to-map-json-data-from-a-rest-api-to-azure-sql-using-data-factory

标签

json

azure

azure-data-factory