Stream Analytics Job -> DataLake ouput

Question

I want to set up CI/CD (ARM template) with StreamAnalytics Job with output set to DataLake Store. https://docs.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs/outputs#microsoftdatalakeaccounts

The issue comes with refreshToken: "It is recommended to put a dummy string value here when creating the data source and then going to the Azure Portal to authenticate the data source which will update this property with a valid refresh token"

Furthermore after 90-days refresh token is outdated and you need to do "Renvew Authorization" https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-data-lake-output#renew-data-lake-store-authorization

I tried to authorize ServicePrincipal.

How to do automatic deployment for ASA with DataLake? How to handle issue with this 90-days token validitiy?

Maybe you wiped the trail :)

Answer 1

at this time it is not yet possible. Sorry for the inconvenience. However we know this is very important and we will add Service Principal auth in the near future (we are looking at the exact ETA).

In the meantime you need to renew manually the token. This can be done without losing any data by (1) stopping the job, (2) changing the token, then (3) restarting the job last time it was stopped.

Let me know if you have any further question.

Answer 2

As far as I know quite soon MSI-based authentication will be in preview.

But if you need an immediate solution (to e.g. be able to have a VSTS pipeline running through without error) you can do the following:

• Create template (e.g. with the CICD NuGet Package [1])

• Manipulate the ARM Template <jobName>.JobTemplate.json

  • Add the output datasource object for the ADLS output object
  • If you work with Visual Studio you can get the values quite easily from the ADLS output JSON
  • It is important to set refreshToken to some fake value

Like the following:

"outputs": [
    {
        "name": "xxx",
        "properties": {
            "serialization": {
                "type": "Json",
                "properties": {
                    "encoding": "UTF8",
                    "format": "LineSeparated"
                }
            },
            "datasource": {
                "type": "Microsoft.DataLake/Accounts",
                "properties": {
                    "accountName": "xxx",
                    "tenantId": "xxx-xxx-xxx-xxx-xxx",
                    "tokenUserPrincipalName": "xxx@xxx.com",
                    "tokenUserDisplayName": "xxx, xxx",
                    "filePathPrefix": "xxx/{date}/{time}",
                    "dateFormat": "yyyy/MM/dd",
                    "timeFormat": "HH",
                    "refreshToken": "faketoken"
                }
            }
        }
    },
    ...

• Deploy the ARM Template

• The job will start successfully but it is necessary to renew the token, therefore

  • Stop the job

  • Renew the authentication of the ADLS output

  • Start the job

Resources

[1] CICD NuGet Package