NUnit Test Run Order

Question

By default nunit tests run alphabetically. Does anyone know of any way to set the execution order? Does an attribute exist for this?

Answer 1

I am working with Selenium WebDriver end-to-end UI test cases written in C#, which are run using NUnit framework. (Not unit cases as such)

These UI tests certainly depend on order of execution, as other test needs to add some data as a precondition. (It is not feasible to do the steps in every test)

Now, after adding the 10th test case, i see NUnit wants to run in this order: Test_1 Test_10 Test_2 Test_3 ..

So i guess i have to too alphabetisize the test case names for now, but it would be good to have this small feature of controlling execution order added to NUnit.

Answer 2

In case of using TestCaseSource the key is to override string ToString method, How that works:

Assume you have TestCase class

public class TestCase
{
    public string Name { get; set; }
    public int Input { get; set; }
    public int Expected { get; set; }
}

And a list of TestCases:

private static IEnumerable<TestCase> TestSource()
{
    return new List<TestCase>
    {
        new TestCase()
        {
           Name = "Test 1",
           Input = 2,
           Expected = 4
        },
        new TestCase()
        {
            Name = "Test 2",
            Input = 4,
            Expected = 16
        },
        new TestCase()
        {
            Name = "Test 3",
            Input = 10,
            Expected = 100
        }
    };
}

Now lets use it with a Test method and see what happen:

[TestCaseSource(nameof(TestSource))]
public void MethodXTest(TestCase testCase)
{
    var x = Power(testCase.Input);
    x.ShouldBe(testCase.Expected);
}

This will not test in order and the output will be like this:

So if we added override string ToString to our class like:

public class TestCase
{
    public string Name { get; set; }
    public int Input { get; set; }
    public int Expected { get; set; }

    public override string ToString()
    {
        return Name;
    }
}

The result will change and we get the order and name of test like:

Note:

1. This is just example to illustrate how to get name and order in test, the order is taken numerically/alphabetically so if you have more than ten test I suggest making Test 01, Test 02.... Test 10, Test 11 etc because if you make Test 1 and at some point Test 10 than the order will be Test 1, Test 10, Test 2 .. etc.
2. The input and Expected can be any type, string, object or custom class.
3. Beside order, the good thing here is that you see the test name which is more important.

Answer 3

I just want to point out that while most of the responders assumed these were unit tests, the question did not specify that they were.

nUnit is a great tool that can be used for a variety of testing situations. I can see appropriate reasons for wanting to control test order.

In those situations I have had to resort to incorporating a run order into the test name. It would be great to be able to specify run order using an attribute.

Answer 4

I am testing with Selenium on a fairly complex web site and the whole suite of tests can run for more than a half hour, and I'm not near to covering the entire application yet. If I have to make sure that all previous forms are filled in correctly for each test, this adds a great deal of time, not just a small amount of time, to the overall test. If there's too much overhead to running the tests, people won't run them as often as they should.

So, I put them in order and depend on previous tests to have text boxes and such completed. I use Assert.Ignore() when the pre-conditions are not valid, but I need to have them running in order.

Answer 5

This question is really old now, but for people who may reach this from searching, I took the excellent answers from user3275462 and PvtVandals / Rico and added them to a GitHub repository along with some of my own updates. I also created an associated blog post with some additional info you can look at for more info.

Hope this is helpful for you all. Also, I often like to use the Category attribute to differentiate my integration tests or other end-to-end tests from my actual unit tests. Others have pointed out that unit tests should not have an order dependency, but other test types often do, so this provides a nice way to only run the category of tests you want and also order those end-to-end tests.

Answer 6

There are very good reasons to utilise a Test Ordering mechanism. Most of my own tests use good practice such as setup/teardown. Others require huge amounts of data setup, which can then be used to test a range of features. Up till now, I have used large tests to handle these (Selenium Webdriver) integration tests. However, I think the above suggested post on https://github.com/nunit/docs/wiki/Order-Attribute has a lot of merit. Here is an example of why ordering would be extremely valuable:

• Using Selenium Webdriver to run a test to download a report
• The state of the report (whether it's downloadable or not) is cached for 10 minutes
• That means, before every test I need to reset the report state and then wait up to 10 minutes before the state is confirmed to have changed, and then verify the report downloads correctly.
• The reports cannot be generated in a practical / timely fashion through mocking or any other mechanism within the test framework due to their complexity.

This 10 minute wait time slows down the test suite. When you multiply similar caching delays across a multitude of tests, it consumes a lot of time. Ordering tests could allow data setup to be done as a "Test" right at the beginning of the test suite, with tests relying on the cache to bust being executed toward the end of the test run.