How to run test methods in specific order in JUnit4?

Question

I want to execute test methods which are annotated by @Test in specific order.

For example:

public class MyTest {
    @Test public void          


        

Answer 1

JUnit 5 update (and my opinion)

I think it's quite important feature for JUnit, if author of JUnit doesn't want the order feature, why?

By default, unit testing libraries don't try to execute tests in the order that occurs in the source file.
JUnit 5 as JUnit 4 work in that way. Why ? Because if the order matters it means that some tests are coupled between them and that is undesirable for unit tests.
So the @Nested feature introduced by JUnit 5 follows the same default approach.

But for integration tests, the order of the test method may matter since a test method may change the state of the application in a way expected by another test method. For example when you write an integration test for a e-shop checkout processing, the first test method to be executed is registering a client, the second is adding items in the basket and the last one is doing the checkout. If the test runner doesn't respect that order, the test scenario is flawed and will fail.
So in JUnit 5 (from the 5.4 version) you have all the same the possibility to set the execution order by annotating the test class with @TestMethodOrder(OrderAnnotation.class) and by specifying the order with @Order(numericOrderValue) for the methods which the order matters.

For example :

@TestMethodOrder(OrderAnnotation.class) 
class FooTest {

    @Order(3)
    @Test
    void checkoutOrder() {
        System.out.println("checkoutOrder");
    }

    @Order(2)
    @Test
    void addItemsInBasket() {
        System.out.println("addItemsInBasket");
    }

    @Order(1)
    @Test
    void createUserAndLogin() {
        System.out.println("createUserAndLogin");
    }
}

Output :

createUserAndLogin

addItemsInBasket

checkoutOrder

By the way, specifying @TestMethodOrder(OrderAnnotation.class) looks like not needed (at least in the 5.4.0 version I tested).

Side note
About the question : is JUnit 5 the best choice to write integration tests ? I don't think that it should be the first tool to consider (Cucumber and co may often bring more specific value and features for integration tests) but in some integration test cases, the JUnit framework is enough. So that is a good news that the feature exists.

Answer 2

Look at a JUnit report. JUnit is already organized by package. Each package has (or can have) TestSuite classes, each of which in turn run multiple TestCases. Each TestCase can have multiple test methods of the form public void test*(), each of which will actually become an instance of the TestCase class to which they belong. Each test method (TestCase instance) has a name and a pass/fail criteria.

What my management requires is the concept of individual TestStep items, each of which reports their own pass/fail criteria. Failure of any test step must not prevent the execution of subsequent test steps.

In the past, test developers in my position organized TestCase classes into packages that correspond to the part(s) of the product under test, created a TestCase class for each test, and made each test method a separate "step" in the test, complete with its own pass/fail criteria in the JUnit output. Each TestCase is a standalone "test", but the individual methods, or test "steps" within the TestCase, must occur in a specific order.

The TestCase methods were the steps of the TestCase, and test designers got a separate pass/fail criterion per test step. Now the test steps are jumbled, and the tests (of course) fail.

For example:

Class testStateChanges extends TestCase

public void testCreateObjectPlacesTheObjectInStateA()
public void testTransitionToStateBAndValidateStateB()
public void testTransitionToStateCAndValidateStateC()
public void testTryToDeleteObjectinStateCAndValidateObjectStillExists()
public void testTransitionToStateAAndValidateStateA()
public void testDeleteObjectInStateAAndObjectDoesNotExist()
public void cleanupIfAnythingWentWrong()

Each test method asserts and reports its own separate pass/fail criteria. Collapsing this into "one big test method" for the sake of ordering loses the pass/fail criteria granularity of each "step" in the JUnit summary report. ...and that upsets my managers. They are currently demanding another alternative.

Can anyone explain how a JUnit with scrambled test method ordering would support separate pass/fail criteria of each sequential test step, as exemplified above and required by my management?

Regardless of the documentation, I see this as a serious regression in the JUnit framework that is making life difficult for lots of test developers.

Answer 3

I think it's quite important feature for JUnit, if author of JUnit doesn't want the order feature, why?

I'm not sure there is a clean way to do this with JUnit, to my knowledge JUnit assumes that all tests can be performed in an arbitrary order. From the FAQ:

How do I use a test fixture?

(...) The ordering of test-method invocations is not guaranteed, so testOneItemCollection() might be executed before testEmptyCollection(). (...)

Why is it so? Well, I believe that making tests order dependent is a practice that the authors don't want to promote. Tests should be independent, they shouldn't be coupled and violating this will make things harder to maintain, will break the ability to run tests individually (obviously), etc.

That being said, if you really want to go in this direction, consider using TestNG since it supports running tests methods in any arbitrary order natively (and things like specifying that methods depends on groups of methods). Cedric Beust explains how to do this in order of execution of tests in testng.

Answer 4

I ended up here thinking that my tests weren't run in order, but the truth is that the mess was in my async jobs. When working with concurrency you need to perform concurrency checks between your tests as well. In my case, jobs and tests share a semaphore, so next tests hang until the running job releases the lock.

I know this is not fully related to this question, but maybe could help targeting the correct issue

Answer 5

you can use one of these piece of codes:

@FixMethodOrder(MethodSorters.JVM)OR `@FixMethodOrder(MethodSorters.DEFAULT)` OR `@FixMethodOrder(MethodSorters.NAME_ASCENDING)` before your test class like this:


@FixMethodOrder(MethodSorters.NAME_ASCENDING)


public class BookTest { ...}

Answer 6

Here is an extension to JUnit that can produce the desired behavior: https://github.com/aafuks/aaf-junit

I know that this is against the authors of JUnit philosophy, but when using JUnit in environments that are not strict unit testing (as practiced in Java) this can be very helpful.