TFS 2017 test task run fails on VS 2017 build machine if code coverage is enabled

Question

We're executing a number of tests on a build machine with VS 2017 Enterprise where we've enabled code coverage on the Visual Studio test task. This fails with the following error in the TFS 2017 console:

##[section]Starting: Test Assemblies **\*test*.dll
Preparing task execution handler.
Executing the powershell script: d:\builds\4\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\1.0.60\VSTest.ps1
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform
Working folder: d:\builds\4\3\s
Executing C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "d:\builds\4\3\s\Source\u.a.testassembly.dll"  /Settings:"C:\Users\username\AppData\Local\Temp\tmp9CCF.tmp" /EnableCodeCoverage /logger:trx /TestAdapterPath:"d:\builds\4\3\s"
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

##[error]The test source file "/EnableCodeCoverage" provided was not found.

With Visual Studio 2015 enterprise the tests were executed with code coverage with no error.

We're supplying the path for the vstest.console.exe file as a custom path.

If I disable code coverage in the checkbox for the TFS 2017 'Visual Studio Test' task and use a runsettings file with no code coverage collectors specified, the tests are executed, but no code coverage is reported.

I've also tried with the code coverage checkbox disabled and the

/EnableCodeCoverage

argument supplied as a commandline parameter on the test tasks. That provides the same error as above.

I've found a number of people having issues with code coverage in VS 2017 but that mostly seems related to .NET core project?

Thank you for your help! :-)

UPDATE 1

If I try to run vstest.console.exe through the developer command prompt I'm unable to successfully specify the /EnableCodeCoverage option. It provides the following error message:

Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

The test source file "/EnableCodeCoverage" provided was not found.

This errormessage does not appear if I run the exact same command without the above switch.

Update 2

If I use a Runsettings file instead, in which I specify the datacollectors, I get a JSON related error:

Cannot deserialize the current JSON object (e.g. {"name":"value"}) 
into type 'System.Collections.ObjectModel.Collection`1 Microsoft.VisualStudio.TestPlatform.ObjectModel.AttachmentSet]' 
because the type requires a JSON array (e.g. [1,2,3]) to deserialize correctly.

If I use the /Diag switch I get the following stacktrace in a logfile:

Stack Trace:
ved Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
ved Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
ved Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
ved Newtonsoft.Json.Linq.JToken.ToObject(Type objectType, JsonSerializer jsonSerializer)
ved Newtonsoft.Json.Linq.JToken.ToObject[T](JsonSerializer jsonSerializer)
ved Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.JsonDataSerializer.DeserializePayload[T](Message message)
ved Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyDataCollectionManager.<>c__DisplayClass6_0.<AfterTestRunEnd>b__0()
ved Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.DataCollection.ProxyDataCollectionManager.InvokeDataCollectionServiceAction(Action action, ITestMessageEventHandler runEventsHandler)

Update 3

Our build server is a Windows 2012 standard server. We were able to install VS 2017 without any errors except for the installation of the Windows SDK payload. We had to manually install Windows SDK for Windows 8 afterwards as Windows SDK payload included in the VS 2017 Enterprise didn't install on Windows server 2012.

I've just noticed that Windows Server 2012 isn't on the list of supported operating systems for VS 2017?...

Update 4

The detailed output from vstest.console.exe /? when run from the VS 2017 developer command prompt is:

Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Usage: vstest.console.exe [Arguments] [Options] [[--] <args>...]]
Description: Runs tests from the specified files.

It then lists all of the arguments. The

/Enablecodecoverage

switch is nowhere to be found. The Developer Command Prompt is opened using the shortcut in the Start menu.

I've tried to run a test suite where the datacollector was defined in a .runsettings file instead of using the /enablecodecoverage switch, but that produces the JSON error listed above.

Update 5

After upgrading our TFS on-premise to 2017 update 1 we we're able to successfully execute our unit tests AND get code coverage.

We are however using SonarQube and apparently at this time of writing the SonarQube scanner provided in the Marketplace Extension v. 3.0.0 doesn't correctly locate the code coverage report due to the change in registry keys done in VS 2017.

A bug report has been filed with SonarQube on this issue.

You can get around this limitation by manually creating a registry value named 'ShellFolder' under the follow registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\15.0

The 'ShellFolder' string value should then be:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise

I've left the above information here in case somebody else comes looking with same problems that we had.

Answer 1

The reason is that there isn’t /EnableCodeCoverage option if call vstest.console.exe tool directly. (Sepecify vstest.console.exe path for Visual Studio Test step/task). If call vstest.console command through Developer Command Prompt for VS 2017, it has /EnableCodeCoverage.

To deal with this issue, you can upgrade your TFS 2017 to TFS 2017 update 1, then select 2.* version of Visual Studio Test step/task, after that you can select Visual Studio 2017 option for VSTest version.

Update:

Another workaround is that, you can install agent for VS 2017 (Tools for Visual Studio 2017) on build agent, then specify vstest.console.exe path in test agent installation folder for Visual Studio Test step/task. (C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe)