Azure KeyVault Active Directory AcquireTokenAsync timeout when called asynchronously

Question

I have setup Azure Keyvault on my ASP.Net MVC web application by following the example in Microsoft\'s Hello Key Vault sample application.

Azure KeyVault (Active Directo

Answer 1

I have the same challenge you have. I am assuming that you've also seen the sample published at https://azure.microsoft.com/en-us/documentation/articles/key-vault-use-from-web-application/

There is a big difference between what that sample does and what my code does (and I think the intent of your code is). In the sample, they retrieve a secrete and store it in the web application as a static member of their Utils class. Thus, the sample retrieves a secret one time for the entire run time of the application.

In my case, I am retrieving a different key for different purposes at different times during the application's run time.

Additionally, the sample download you linked to uses an X.509 certificate to authenticate the web application to KeyVault, rather than client secret. It's possible there is an issue with that too.

I saw the chat with @shaun-luttin concluding you caused a deadlock, but that's not the whole story I think. I don't use .GetAwaiter().GetResult() or call an async method from a ctor.

Answer 2

Problem: deadlock

Your EncryptionProvider() is calling GetAwaiter().GetResult(). This blocks the thread, and on subsequent token requests, causes a deadlock. The following code is the same as yours is but separates things to facilitate explanation.

public AzureEncryptionProvider() // runs in ThreadASP
{
    var client = new KeyVaultClient(GetAccessToken);

    var task = client.GetKeyAsync(KeyVaultUrl, KeyVaultEncryptionKeyName);

    var awaiter = task.GetAwaiter();

    // blocks ThreadASP until GetKeyAsync() completes
    var keyBundle = awaiter.GetResult();
}

In both token requests, the execution starts in the same way:

• AzureEncryptionProvider() runs in what we'll call ThreadASP.
• AzureEncryptionProvider() calls GetKeyAsync().

Then things differ. The first token request is multi-threaded:

1. GetKeyAsync() returns a Task.
2. We call GetResult() blocking ThreadASP until GetKeyAsync() completes.
3. GetKeyAsync() calls GetAccessToken() on another thread.
4. GetAccessToken() and GetKeyAsync() complete, freeing ThreadASP.
5. Our web page returns to the user. Good.

The second token request uses a single thread:

1. GetKeyAsync() calls GetAccessToken() on ThreadASP (not on a separate thread.)
2. GetKeyAsync() returns a Task.
3. We call GetResult() blocking ThreadASP until GetKeyAsync() completes.
4. GetAccessToken() must wait until ThreadASP is free, ThreadASP must wait until GetKeyAsync() completes, GetKeyAsync() must wait until GetAccessToken() completes. Uh oh.
5. Deadlock.

Why? Who knows?!?

There must be some flow control within GetKeyAsync() that relies on the state of our access token cache. The flow control decides whether to run GetAccessToken() on its own thread and at what point to return the Task.

Solution: async all the way down

To avoid a deadlock, it is a best practice "to use async all the way down." This is especially true when we are calling an async method, such as GetKeyAsync(), that is from an external library. It is important not force the method to by synchronous with Wait(), Result, or GetResult(). Instead, use async and await because await pauses the method instead of blocking the whole thread.

Async controller action

public class HomeController : Controller
{
    public async Task<ActionResult> Index()
    {
        var provider = new EncryptionProvider();
        await provider.GetKeyBundle();
        var x = provider.MyKeyBundle;
        return View();
    }
}

Async public method

Since a constructor cannot be async (because async methods must return a Task), we can put the async stuff into a separate public method.

public class EncryptionProvider
{
    //
    // authentication properties omitted

    public KeyBundle MyKeyBundle;

    public EncryptionProvider() { }

    public async Task GetKeyBundle()
    {
        var keyVaultClient = new KeyVaultClient(GetAccessToken);
        var keyBundleTask = await keyVaultClient
            .GetKeyAsync(KeyVaultUrl, KeyVaultEncryptionKeyName);
        MyKeyBundle = keyBundleTask;
    }

    private async Task<string> GetAccessToken(
        string authority, string resource, string scope)
    {
        TokenCache.DefaultShared.Clear(); // reproduce issue 
        var authContext = new AuthenticationContext(authority, TokenCache.DefaultShared);
        var clientCredential = new ClientCredential(ClientIdWeb, ClientSecretWeb);
        var result = await authContext.AcquireTokenAsync(resource, clientCredential);
        var token = result.AccessToken;
        return token;
    }
}

Mystery solved. :) Here is a final reference that helped my understanding.

Console App

My original answer had this console app. It worked as an initial troubleshooting step. It did not reproduce the problem.

The console app loops every five minutes, repeatedly asking for a new access token. At each loop, it outputs the current time, the expiry time, and the name of the retrieved key.

On my machine, the console app ran for 1.5 hours and successfully retrieved a key after expiration of the original.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Azure.KeyVault;
using Microsoft.IdentityModel.Clients.ActiveDirectory;

namespace ConsoleApp
{
    class Program
    {
        private static async Task RunSample()
        {
            var keyVaultClient = new KeyVaultClient(GetAccessToken);

            // create a key :)
            var keyCreate = await keyVaultClient.CreateKeyAsync(
                vault: _keyVaultUrl,
                keyName: _keyVaultEncryptionKeyName,
                keyType: _keyType,
                keyAttributes: new KeyAttributes()
                {
                    Enabled = true,
                    Expires = UnixEpoch.FromUnixTime(int.MaxValue),
                    NotBefore = UnixEpoch.FromUnixTime(0),
                },
                tags: new Dictionary<string, string> {
                    { "purpose", "StackOverflow Demo" }
                });

            Console.WriteLine(string.Format(
                "Created {0} ",
                keyCreate.KeyIdentifier.Name));

            // retrieve the key
            var keyRetrieve = await keyVaultClient.GetKeyAsync(
                _keyVaultUrl,
                _keyVaultEncryptionKeyName);

            Console.WriteLine(string.Format(
                "Retrieved {0} ",
                keyRetrieve.KeyIdentifier.Name));
        }

        private static async Task<string> GetAccessToken(
            string authority, string resource, string scope)
        {
            var clientCredential = new ClientCredential(
                _keyVaultAuthClientId,
                _keyVaultAuthClientSecret);

            var context = new AuthenticationContext(
                authority,
                TokenCache.DefaultShared);

            var result = await context.AcquireTokenAsync(resource, clientCredential);

            _expiresOn = result.ExpiresOn.DateTime;

            Console.WriteLine(DateTime.UtcNow.ToShortTimeString());
            Console.WriteLine(_expiresOn.ToShortTimeString());

            return result.AccessToken;
        }

        private static DateTime _expiresOn;
        private static string
            _keyVaultAuthClientId = "xxxxx-xxx-xxxxx-xxx-xxxxx",
            _keyVaultAuthClientSecret = "xxxxx-xxx-xxxxx-xxx-xxxxx",
            _keyVaultEncryptionKeyName = "MYENCRYPTIONKEY",
            _keyVaultUrl = "https://xxxxx.vault.azure.net/",
            _keyType = "RSA";

        static void Main(string[] args)
        {
            var keepGoing = true;
            while (keepGoing)
            {
                RunSample().GetAwaiter().GetResult();
                // sleep for five minutes
                System.Threading.Thread.Sleep(new TimeSpan(0, 5, 0)); 
                if (DateTime.UtcNow > _expiresOn)
                {
                    Console.WriteLine("---Expired---");
                    Console.ReadLine();
                }
            }
        }
    }
}