Has JUnit4 begun supporting ordering of test? Is it intentional?

Question

A newbie to JUnit (in fact JUnit 4) and came across suite way of executing test

@RunWith(Suite.class)
@Suite.SuiteClasses(
        {                                  


        

Answer 1

Just bypass it by ordering the names of the class by using an _# before the class name. Ex _1testcaselogin _2testcaseverify

Answer 2

@jeha answer is very good but he talks about test methods (the ones annotated with @Test) order. This order is indeed unspecified, probably because the order of methods in a class obtained via reflection is not guaranteed to correspond with the order of methods in the source file.

However from my experience they are executed in the same order they were defined most of the time.

You are asking about the order of test classes. This should not be a surprise: you are explicitly listing test classes using @Suite.SuiteClasses and JUnit probably has no business in shuffling them and running in different order. Are you concerned about introducing dependency between tests simply because you explicitly ordered them?

Answer 3

JUnit 4.11 now supports specifying execution order using @FixMethodOrder annoation.

Answer 4

No, ordering is not supported. And yes, it is intention:

KentBeck commented (December 30, 2010): Independent tests are more valuable, so JUnit by design does not support test ordering.

If you need this feature, you could use TestNG with it's @Test(sequential = true).

Answer 5

No JUnit does not support test ordering, except in the manner that you say, through Suite. This only defines the order in which the test classes are executed. This has been there for a long time, including JUnit 3 Suite classes.

For a more complete explanation, there are three things we need to talk about here:

1. the ordering of tests classes within a test suite
2. the ordering of test classes when they are found by reflection by Eclipse or Maven
3. the ordering of test methods (annotated with @Test) within a test class.

The ordering of tests classes within a test suite

When you specify the list of classes to be executed in a test suite, you're defining an array, and these test classes will be executed in order, except when you're doing parallel execution. Unfortunately, this allows the introduction of dependencies between your test classes.

The ordering of test classes when they are found by reflection

When searching the classpath for classes, the order in which they are found is not guaranteed, so cannot be depended upon. It isn't actually JUnit which is doing the searching, but the Eclipse Junit plugin, or maven surefire or failsafe.

The ordering of test methods within a test class

JUnit does not guarantee the order of execution of tests within a class. Most of the time, on most JVMs pre version 7, the order in which they are found using reflection is in declaration order, ie the order they are in the file. This is the order in which they are executed. However, with JVM 7, this is no longer guaranteed, so there won't be a consistent order. There is a github issue #293 Sort test methods for predictability open with suggested solutions, and there is a thread on the junit mailing list: Alphabetizing test method run order?. So you cannot depend upon the order that tests will be executed using JUnit, but this is currently under discussion.

Answer 6

You can extend the Suite runner:

public class MySuite extends Suite {

    public SuiteRunner(Class<?> klass, RunnerBuilder builder) throws InitializationError {
        super(klass, builder);
    }

    @Override
    protected List<Runner> getChildren() {
        List<Runner> children = super.getChildren();
        ... here modify the children list - you can remove or reorder test...
        return children;
    }
}

Then annotate your suite:

@RunWith(MySuite.class)
@Suite.SuiteClasses({                               
        CreateNewProfile.class,
        EditProfile.class})
public class ProfileTestSuite { }

The tests marked with @Ignore will be represented by IgnoredClassRunner and regular test will be represented by BlockJUnit4ClassRunner (or whatever runner you will annotate them - which will be probably extension of BlockJUnit4ClassRunner).

From the runner you can get the test class. Then you can use own annotations.