JUnit test class order

问题

I have a java app with maven. Junit for tests, with failsafe and surefire plugins. I have more than 2000 integration tests. To speed up the test running, I use failsafe jvmfork to run my tests parallel. I have some heavy test class, and they typically running at end of my test execution and it is slows down my CI verify process. The filesafe runorder:balanced would be a good option for me, but i cant use it because the jvmfork. To rename the test classes or move to another package and run it alpahabetical is not an option. Any suggestion how can I run my slow test classes at the begining of the verify process?

回答1:

In out project we had created a few marker interfaces ( example

public interface SlowTestsCategory {}

)

and put it into the @Category annotation of JUnit in the test class with slow tests.

@Category(SlowTestsCategory.class)

After that we created some special tasks for Gradle to run tests by category or a few categories by custom order:

task unitTest(type: Test) {
  description = 'description.'
  group = 'groupName'

  useJUnit {
    includeCategories 'package.SlowTestsCategory'
    excludeCategories 'package.ExcludedCategory'
  }
}

This solution is served by Gradle, but maybe it'll be helpful for you.

回答2:

I gave the combination of answers I found a try:

Running JUnit4 Test classes in specified order
Running JUnit Test in parallel on Suite Level

The second answer is based on these classes of this github project, which is available under the BSD-2 license.

I defined a few test classes:

public class LongRunningTest {

    @Test
    public void test() {

        System.out.println(Thread.currentThread().getName() + ":\tlong test - started");

        long time = System.currentTimeMillis();
        do {
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
            }
        } while(System.currentTimeMillis() - time < 1000);

        System.out.println(Thread.currentThread().getName() + ":\tlong test - done");
    }
}

@Concurrent
public class FastRunningTest1 {

    @Test
    public void test1() {
        try {
            Thread.sleep(250);
        } catch (InterruptedException e) {
        }

        System.out.println(Thread.currentThread().getName() + ":\tfrt1-test1 - done");
    }

    // +7 more repetions of the same method
}

Then I defined the test suites:
(FastRunningTest2 is a copy of the first class with adjusted output)

@SuiteClasses({LongRunningTest.class, LongRunningTest.class})
@RunWith(Suite.class)
public class SuiteOne {}

@SuiteClasses({FastRunningTest1.class, FastRunningTest2.class})
@RunWith(Suite.class)
public class SuiteTwo {}

@SuiteClasses({SuiteOne.class, SuiteTwo.class})
@RunWith(ConcurrentSuite.class)
public class TopLevelSuite {}

When I execute the TopLevelSuite I get the following output:

TopLevelSuite-1-thread-1: long test - started FastRunningTest1-1-thread-4: frt1-test4 - done FastRunningTest1-1-thread-2: frt1-test2 - done FastRunningTest1-1-thread-1: frt1-test1 - done FastRunningTest1-1-thread-3: frt1-test3 - done FastRunningTest1-1-thread-5: frt1-test5 - done FastRunningTest1-1-thread-3: frt1-test6 - done FastRunningTest1-1-thread-1: frt1-test8 - done FastRunningTest1-1-thread-5: frt1-test7 - done FastRunningTest2-2-thread-1: frt2-test1 - done FastRunningTest2-2-thread-2: frt2-test2 - done FastRunningTest2-2-thread-5: frt2-test5 - done FastRunningTest2-2-thread-3: frt2-test3 - done FastRunningTest2-2-thread-4: frt2-test4 - done TopLevelSuite-1-thread-1: long test - done TopLevelSuite-1-thread-1: long test - started FastRunningTest2-2-thread-5: frt2-test8 - done FastRunningTest2-2-thread-2: frt2-test6 - done FastRunningTest2-2-thread-1: frt2-test7 - done TopLevelSuite-1-thread-1: long test - done

Which basically shows that the LongRunningTest is executed in parralel to the FastRunningTests. The default value of threads used for parallel execution defined by the Concurrent Annotation is 5, which can be seen in the output of the parallel execution of the FastRunningTests.

The downside is that theses Threads are not shared between FastRunningTest1 and FastRunningTest2.

This behavious shows that it is "somewhat" possible to do what you want to do (so whether that works with your current setup is a different question).

Also I am not sure whether this is actually worth the effort,

as you need to prepare those TestSuites manually (or write something that autogenerates them)
and you need to define the Concurrent Annotation for all those classes (maybe with a different number of threads for each class)

As this basically shows that it is possible to define the execution order of classes and trigger their parallel execution, it should also be possibly to get the whole process to only use one ThreadPool (but I am not sure what the implication of that would be).

As the whole concept is based on a ThreadPoolExecutor, using a PriorityBlockingQueue which gives long running tasks a higher priority you would get closer to your ideal outcome of executing the long running tests first.

I experimented around a bit more and implemented my own custom suite runner and junit runner. The idea behind is to have your JUnitRunner submit the tests into a queue which is handeld by a single ThreadPoolExecutor. Because I didn't implement a blocking operation in the RunnerScheduler#finish method, I ended up with a solution where the tests from all classes were passed to the queue before the execution even started. (That might look different if there a more test classes and methods involved).

At least it proves the point that you can mess with junit at this level if you really want to.

The code of my poc is a bit messy and to lengthy to put it here, but if someone is interested I can push it into a github project.

回答3:

Let me summarize everything before I will provide a recommendation.

Integration tests are slow. This is fine and it's natural.
CI build doesn't run tests that assume deployment of a system, since there is no deployment in CI. We care about deployment in CD process. So I assume your integration tests don't assume deployment.
CI build runs unit tests first. Unit tests are extremely fast because they use only RAM.
We have good and quick feedback from unit tests.

At this moment we are sure we don't have a problem with getting a quick feedback. But we still want to run integration tests faster. I would recommend the following solutions:

Improve actual tests. Quite often they are not effective and can be speed up significantly.
Run integration tests in background (i.e. don't wait for real time feedback from them).
It's natural for them to be much slower than unit tests.
Split integration tests on groups and run them separately if you need feedback from some of them faster.
Run integration tests in different JVMs. Not different threads within the same JVM!
In this case you don't care about thread safety and you should not care about it.
Run integration tests on different machines and so on.

I worked with many different projects (some of them had CI build running for 48 hours) and first 3 steps were enough (even for crazy cases). Step #4 is rarely needed having good tests. Step #5 is for very specific situations.

You see that my recommendation relates to the process and not to the tool, because the problem is in the process.
Quite often people ignore root cause and try to tune the tool (Maven in this case). They get cosmetic improvements but with high maintenance cost of created solution.

回答4:

In JUnit 5 (from version 5.8.0-M1 onwards) test classes can be ordered too.

src/test/resources/junit-platform.properties:

# ClassOrderer$OrderAnnotation sorts classes based on their @Order annotation
junit.jupiter.testclass.order.default=org.junit.jupiter.api.ClassOrderer$OrderAnnotation

Other Junit built-in class orderer implementations:

org.junit.jupiter.api.ClassOrderer$ClassName
org.junit.jupiter.api.ClassOrderer$DisplayName
org.junit.jupiter.api.ClassOrderer$Random

^{For other ways (beside junit-platform.properties file) to set configuration parameters refer here.}

You can also provide your own orderer. It must implement ClassOrderer interface:

package foo;
public class MyOrderer implements ClassOrderer {
    @Override
    public void orderClasses(ClassOrdererContext context) {
        Collections.shuffle(context.getClassDescriptors());
    }
}

junit.jupiter.testclass.order.default=foo.MyOrderer

Note that `@Nested` test classes cannot be ordered by a `ClassOrderer`.

Refer to JUnit 5 documentations and ClassOrderer api docs to learn more about this.

回答5:

You can use annotations in Junit 5 to set the test order you wish to use:

From Junit 5's user guide: https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order

import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(OrderAnnotation.class)
class OrderedTestsDemo {

    @Test
    @Order(1)
    void nullValues() {
        // perform assertions against null values
    }

    @Test
    @Order(2)
    void emptyValues() {
        // perform assertions against empty values
    }

    @Test
    @Order(3)
    void validValues() {
        // perform assertions against valid values
    }

}

Upgrading to Junit5 can be done fairly easily and the documentation on the link in the beginning of the post contains all the information you might need.

来源：https://stackoverflow.com/questions/57624495/junit-test-class-order

标签

java