Is there anything I can do in NUnit that I can't do in MSTest?

Question

This question has been asked in various forms in a number of different forums, but, IMHO, I haven\'t been able to find a place where it\'s really answered clearly, so I\'m g

Answer 1

@Dave Hanna MS Test is available out of the box so one less component to worry about deploying and keeping updated. It is also integrated with Visual Studio and other Microsoft products by design.

The fluent syntax is nice but it's not a functional difference so I'd ignore it. Not to mention that you can have the same in MS Test by using an extension library.

Performance. 99% of test performance is controlled by system under test, so even a 100% performance difference between two library would result in an ignorable overall difference in performance.

Open source vs. not open source: the chances that you'll benefit from using an open source library are minimal. The exceptions are when changes are pulled into trunk frequently or when there are enough resources to keep the modified branch updated.

Answer 2

• NUnit contains a [TestCase] attribute that allows implementing parametrized tests. This does not exist out of the box in MSTest - it can be done via extensibility though.
• MsTest's ExpectedException attribute has a bug where the expected message is never really asserted even if it's wrong - the test will pass.
• NUnit ships with an Assert.Throws API to allow testing an exception on a specific line of code instead of the whole method. A similar feature exists for MSTest (implemented by the same person who did it for NUnit) but does not ship with MSTest.
• NUnit contains a fluent version of Assert API out of the box. MSTest has third party extensions that do this, but none are shipped with MSTest.
• NUnit allows abstract classes to be test fixtures (so you can inherit test fixtures). MsTest allows this but limits abstract classes to a single assembly.
• NUnit allows non public classes to be test fixtures (as of the latest version)
• NUnit was created SOLELY for the idea of unit testing. MSTest was created for Testing - and also a bit of unit testing.
• NUnit contains PNunit (running parallel tests with NUnit). MSTest added this ability in Visual Studio 2010 which is configurable via XML

Answer 3

NUnit has a richer assert API. The api is particularly elegant (fluent, even), for example

Assert.That(Is.Unique, myResults);  // assert: myResults is a collection of unique items

If you've seen the Hamcrest extensions to JUnit you'll recognise this style.

It also has a growing set of extensions, such as performance testing and an excellent VS plugin.

Answer 4

1. The MSTest test runner is non-deterministic, which should be enough to scare you away from using it. I cannot consistently run MSTest from TFS 2010 using the integrated test runner; it breaks with a few different error messages (and, inconsistently) across project builds and across build agents.
2. MSTest sometimes (not consistently) breaks because it leaks memory--out of memory exceptions still happen for us, even with the newest version of Visual Studio. The workaround for this problem is horrible.
3. I have other minor quibbles with MSTest, and have blogged a bunch of workarounds for using MSTest in general: http://www.pseale.com/blog/TFSAsYourBuildCIServerOnlyPositiveTakeaways1Of2.aspx

I found this question as I am switching us to NUnit from MSTest because we can't trust the result of our CI build anymore because of MSTest. Switching to NUnit will allow us to (finally) trust that a CI build failure is real, not another MSTest glitch. This is the real answer--only with NUnit (or some other non-MSTest test runner) can we trust our CI build.

Answer 5

I've been working on first class support for MSTest in TestDriven.Net 3.0. In previous versions the MSTest support was very basic (there seemed to be few people using MSTest).

As of TestDriven.Net 3.0 Beta 2, there is pretty comprehensive support for all of the unit testing related attributes of MSTest. There is even support for data driven tests using the DataSource attribute. Your tests will also execute at close to NUnit speeds!

If you use MSTest, I'd be interested to hear if any of your unit tests fail when executed using TestDriven.Net 3.0 Beta 2 (or later).

Please try it on your MSTest projects and let me know how you get on: http://www.testdriven.net/download.aspx

NOTE, it doesn't support the DeploymentItem or HostType attributes. You can use the 'Copy to Output Directory' project item setting instead of DeploymentItem.

Answer 6

I can point you to a couple of blogs on frustrations with MSTest:

• MSBuild, NAnt, NUnit, MSTest, and frustration
• Clean build server with MSTest == FAIL

To be fair, these people are trying to set up MSTest on non-TFS build servers. Some of their issues are not going to apply to your situation.

We are primarily a Microsoft shop, and use TFS for source control. However, we use TeamCity for Continuous Integration; we like it, and it integrates reasonably well with TFS. I've never used MSTest; we've been using NUnit for years, and have seen no reason to change.

MSTest is supposed to have tight integration with Team Suite, which (since your company has already paid the outrageous fee for that) is a point in its favor.

NUnit comes with less vendor lock-in, and has a rich API. As serg10 pointed out, the Assert.That syntax is particularly powerful and elegant.

In the end, you can write good unit tests without all the fancy features. Some of them may even get in the way (which is the theory behind xUnit.net). I would recommend that your team standardize on one test framework; avoid having some code in MSTest and other code in NUnit.

I think writing good tests is more important than your choice of frameworks. Consider reading The Art of Unit Testing: with Examples in .NET, writing some tests, then seeing if MSTest is adequate for your team's needs.

EDIT: Appendix B of The Art of Unit Testing has some good comments on Microsoft's Unit Testing Framework. It mentions YUnit as an example of how cumbersome it is to extend MSTest. However, the author does suggest MSTest for Team System users due to the tight integration.