Assert.Inconclusive and IgnoreAttribute

问题

What is the right way to use Assert.Inconclusive and IgnoreAttribute in MS Unit test framework?

We are using Assert.Inconclusive mainly for tests which are:

Not implemented yet
Somehow broken or incomplete = requires futher attention
When test body is for any reason commented out

We are doing this because:

Inconclusive test can have message
We want to see such tests in test results on TFS

Our problem is that Inconclusive tests are marked as error in both TFS and Resharper. If we use IgnoreAttribute instead we will see these tests in Resharper but MS Test runner and TFS will ignore them at all. Using IgnoreAttribute in TFS and MS Test runner is same like commenting whole test which is useless.

回答1:

I like to think how you are describing Inconclusive is the proper usage.

Though in my experience, Inconclusive is treated more like a warning than an error. In fact, they are reported in the TRX separately from errors:

<TestRun>
   <ResultSummary outcome="Inconclusive">
      <Counters total="1" 
          executed="0" error="0" failed="0" 
          timeout="0" aborted="0" inconclusive="1" 
          passedButRunAborted="0" notRunnable="0" 
          notExecuted="0" disconnected="0" 
          warning="0" passed="0" completed="0" 
          inProgress="0" pending="0" />

I typically run the mstest executable from an <Exec> task in my msbuild script and then peek inside the TRX to determine whether it should fail the build.

回答2:

i also see a dilemma in the current implementation.

Inconclusive assertions are included in the TRX report, but mstest.exe (and also vstest.console.exe) will return 1 (meaning error) after execution.
TestMethods with the Ignore attribute will not be reported as an error, but they are completely hidden from the TRX report.

my personal understanding is as follows:

use the [Ignore] attribute to (temporarily) disable / skip the method:

[TestMethod]
[Ignore] // <== disabled through "Ignore" attribute
public void Test001()
{
   //execute some stuff ...
   Assert.IsTrue(...);

   //execute some stuff ...
   Assert.AreEqual(...);
}

do not misuse the Inconclusive assertion for this purpose:

[TestMethod]
public void Test002()
{
    Assert.Inconclusive(); // <== misuse of "Inconclusive" to skip this test

    //execute some stuff ...
}

instead, Inconclusive should be used conditionally: only if we can't tell whether the component to be tested does work as expected or not.
for example in case an external resource we depend on is not available at time of test execution:

[TestMethod]
public void Test003()
{
    //check if the server is running,
    //otherwise can can't test our local client component!
    if (!WebServiceAvailable())
    {
        Assert.Inconclusive(); // <== skip remaining code because the resource is not available
    }

    //execute some stuff ...
    Assert.AreEqual(...);

    //execute some stuff ...
    Assert.AreEqual(...);
}

_ _

conclusion:
to disable / skip a test the logical way is to use the [Ignore] attribute.
i clearly see the current behaviour of mstest.exe not reporting on any ignored test as a bug that should be fixed.

feel free to up-vote the following bug reports:

https://connect.microsoft.com/VisualStudio/feedback/details/779743/
http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2088845

回答3:

I've been doing some research into this, as well as trying it out at home. The end result is that I believe the [Ignore] attribute for MSTest does indeed leave out the test method completely. I tried looking at settings in Visual Studio to see if there was an override, but could not find anything.

Shame about that, as not seeing the ignored tests is bad since you may end up thinking you have a suite of 100 MSTest tests running nicely, but it turns out that there are 50 which are missing from the results that you never knew about due to the [Ignore] attribute! Urgh.

回答4:

Classically, in when testing, unit tests should be very specific so that there is a (close to) one-to-one correspondence between features and tests for these features.

To test certain features, the system under test (SUT) first needs to be brought into a certain state. Once that state is reached, the test can actually test the feature that it is meant for. Now, what should be the status of such a test, if already the setup part fails. It cannot make a statement on the functioning of the feature under test, as the test never reached the status requires to exercise the feature.

In such cases, it is common to assign an inconclusive result, as we simply cannot know if the feature works as it should, and so we cannot pass or fail. The fact that the setup itself does not work as expected should be covered by a separate test.

So imagine, I want to test that a 'foo' that has been 'bar'ed will return a 0 when 'qux'ed. This test should not test if 'foo' can be 'bar'ed, so any failure to be 'bar'ed will return an inconclusive, whereas a failure to respond to 'qux' will be proper fail.

回答5:

As of the MSDN docs:

IgnoreAttribute (since VS 2005) means "this test should not be run" see https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.ignoreattribute(v=vs.80).aspx.
Assert.Inconclusive (since VS 2005) means "Indicates that an assertion cannot be proven true or false. Also used to indicate an assertion that has not yet been implemented." see https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.assert.inconclusive(v=vs.80).aspx

I think these are realy clear statements when you have to use which one.

来源：https://stackoverflow.com/questions/4733592/assert-inconclusive-and-ignoreattribute

标签

.net

unit-testing

resharper

mstest

team-build