Code coverage for async methods

Question

When I analyse code coverage in Visual Studio 2012, any of the await lines in async methods are showing as not covered even though they are obviously executing since my test

Answer 1

This can happen most commonly if the operation you're awaiting is completed before it's awaited.

I recommend you test at least synchronous and asynchronous success situations, but it's also a good idea to test synchronous and asynchronous errors and cancellations.

Answer 2

There are situations where I don't care about testing the async nature of a method but just want to get rid of the partial code coverage. I use below extension method to avoid this and it works just fine for me.

Warning "Thread.Sleep" used here!

public static IReturnsResult<TClass> ReturnsAsyncDelayed<TClass, TResponse>(this ISetup<TClass, Task<TResponse>> setup, TResponse value) where TClass : class
{
    var completionSource = new TaskCompletionSource<TResponse>();
    Task.Run(() => { Thread.Sleep(200); completionSource.SetResult(value); });
    return setup.Returns(completionSource.Task);
}

and the usage is similar to the Moq's ReturnsAsync Setup.

_sampleMock.Setup(s => s.SampleMethodAsync()).ReturnsAsyncDelayed(response);

Answer 3

I created a test runner that runs a block of code multiple times and varies the task that is delayed using a factory. This is great for testing the different paths through simple blocks of code. For more complex paths you may want to create a test per path.

[TestMethod]
public async Task ShouldTestAsync()
{
    await AsyncTestRunner.RunTest(async taskFactory =>
    {
        this.apiRestClient.GetAsync<List<Item1>>(NullString).ReturnsForAnyArgs(taskFactory.Result(new List<Item1>()));
        this.apiRestClient.GetAsync<List<Item2>>(NullString).ReturnsForAnyArgs(taskFactory.Result(new List<Item2>()));

        var items = await this.apiController.GetAsync();

        this.apiRestClient.Received().GetAsync<List<Item1>>(Url1).IgnoreAwait();
        this.apiRestClient.Received().GetAsync<List<Item2>>(Url2).IgnoreAwait();

        Assert.AreEqual(0, items.Count(), "Zero items should be returned.");
    });
}

public static class AsyncTestRunner
{
    public static async Task RunTest(Func<ITestTaskFactory, Task> test)
    {
        var testTaskFactory = new TestTaskFactory();
        while (testTaskFactory.NextTestRun())
        {
           await test(testTaskFactory);
        }
    }
}

public class TestTaskFactory : ITestTaskFactory
{
    public TestTaskFactory()
    {
        this.firstRun = true;
        this.totalTasks = 0;
        this.currentTestRun = -1;   // Start at -1 so it will go to 0 for first run.
        this.currentTaskNumber = 0;
    }

    public bool NextTestRun()
    {
        // Use final task number as total tasks.
        this.totalTasks = this.currentTaskNumber;

        // Always return has next as turn for for first run, and when we have not yet delayed all tasks.
        // We need one more test run that tasks for if they all run sync.
        var hasNext = this.firstRun || this.currentTestRun <= this.totalTasks;

        // Go to next run so we know what task should be delayed, 
        // and then reset the current task number so we start over.
        this.currentTestRun++;
        this.currentTaskNumber = 0;
        this.firstRun = false;

        return hasNext;
    }

    public async Task<T> Result<T>(T value, int delayInMilliseconds = DefaultDelay)
    {
        if (this.TaskShouldBeDelayed())
        {
            await Task.Delay(delayInMilliseconds);
        }

        return value;
    }

    private bool TaskShouldBeDelayed()
    {
        var result = this.currentTaskNumber == this.currentTestRun - 1;
        this.currentTaskNumber++;
        return result;
    }

    public async Task VoidResult(int delayInMilliseconds = DefaultDelay)
    {
        // If the task number we are on matches the test run, 
        // make it delayed so we can cycle through them.
        // Otherwise this task will be complete when it is reached.
        if (this.TaskShouldBeDelayed())
        {
            await Task.Delay(delayInMilliseconds);
        }
    }

    public async Task<T> FromResult<T>(T value, int delayInMilliseconds = DefaultDelay)
    {
        if (this.TaskShouldBeDelayed())
        {
            await Task.Delay(delayInMilliseconds);
        }

        return value;
    }
}

Answer 4

The reason the code is not shown as being covered has to do with how async methods are implemented. The C# compiler actually translates the code in async methods into a class that implements a state machine, and transforms the original method into a stub that initialized and invokes that state machine. Since this code is generated in your assembly, it is included in the code coverage analysis.

If you use a task that is not complete at the time the code being covered is executing, the compiler-generated state machine hooks up a completion callback to resume when the task completes. This more completely exercises the state machine code, and results in complete code coverage (at least for statement-level code coverage tools).

A common way to get a task that is not complete at the moment, but will complete at some point is to use Task.Delay in your unit test. However, that is generally a poor option because the time delay is either too small (and results in unpredictable code coverage because sometimes the task is complete before the code being tests runs) or too large (unnecessarily slowing the tests down).

A better option is to use "await Task.Yield()". This will return immediately but invoke the continuation as soon as it is set.

Another option - though somewhat absurd - is to implement your own awaitable pattern that has the semantics of reporting incomplete until a continuation callback is hooked up, and then to immediately complete. This basically forces the state machine into the async path, providing the complete coverage.

To be sure, this is not a perfect solution. The most unfortunate aspect is that it requires modification to production code to address a limitation of a tool. I would much prefer that the code coverage tool ignore the portions of the async state machine that are generated by the compiler. But until that happens, there aren’t many options if you really want to try to get complete code coverage.

A more complete explanation of this hack can be found here: http://blogs.msdn.com/b/dwayneneed/archive/2014/11/17/code-coverage-with-async-await.aspx