How to make sure TestNG runs methods on test classes in succession instead of interleaved?

Question

The situation and the problem

I have several test classes, each with several test methods. All tests use the same test database in the background. Each test class

Answer 1

I wrote an article discussing some ways to sequence tests in TestNG:

http://ancient-marinator.blogspot.com/2013/05/on-testing-scheduling-your-tests.html

Naturally, the best source for information is at home: http://testng.org/doc/index.html

Answer 2

We were running into this same issue, most people say it's caused by using dependsOn, but our solution was just setting priorities at a test level for some of our tests. I set up a Test Listener to re-prioritize our tests so that they run in the correct order. This is based on saberduck's solution in https://github.com/cbeust/testng/issues/106

This solution will preserve the test priority by concatenating it to the class priority.

package testng_Listeners;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import java.util.HashMap;
import org.testng.IAnnotationTransformer;
import org.testng.Reporter;
import org.testng.annotations.ITestAnnotation;

//Listener to fix TestNG Interleaving issue. I had to re-write this as the original example I had did not allow for priority to be manually set on a test level.
public class RePrioritizingListener implements IAnnotationTransformer {

HashMap<Object, Integer> priorityMap = new HashMap<Object, Integer>();
Integer class_priorityCounter = 10000;
// The length of the final priority assigned to each method.
Integer max_testpriorityLength = 4;

@Override
public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {


// class of the test method.
Class<?> declaringClass = testMethod.getDeclaringClass();
// Current priority of the test assigned at the test method.
Integer test_priority = annotation.getPriority();
// Current class priority.
Integer current_ClassPriority = priorityMap.get(declaringClass);

if (current_ClassPriority == null) {
    current_ClassPriority = class_priorityCounter++;
    priorityMap.put(declaringClass, current_ClassPriority);
}

String concatenatedPriority = test_priority.toString();

// Adds 0's to start of this number.
while (concatenatedPriority.length() < max_testpriorityLength) {
    concatenatedPriority = "0" + concatenatedPriority;
}

// Concatenates our class counter to the test level priority (example
// for test with a priority of 1: 1000100001; same test class with a
// priority of 2: 1000100002; next class with a priority of 1. 1000200001)
concatenatedPriority = current_ClassPriority.toString() + concatenatedPriority;

//Sets the new priority to the test method.
annotation.setPriority(Integer.parseInt(concatenatedPriority));

String printText = testMethod.getName() + " Priority = " + concatenatedPriority;
Reporter.log(printText);
System.out.println(printText);

}
}

Also you'll need to add the listener to your testng.xml

<suite name="Suite" configfailurepolicy="continue" >
<listeners>
<listener class-name="testng_Listeners.RePrioritizingListener"></listener>
</listeners>

Answer 3

I wouldn't use dependsOnGroups at the class level because if any test method in any of the classes you're depending upon fails it won't run your class at all... This is the real downside of using 'dependsOn' groups (or methods). I would try first just setting @Test(group = thisClassName) at the class level, then identify the class with a test tag in your testng.xml file. Then control the ordering of your class runs in the xml as a list of these tests. I believe you may need to also set PreserveOrder = "True" on the next higher level tag in the xml. I would avoid using dependsOn except when you really need it for dependancies, but not for controlling order. Hope this helps. -JR

Answer 4

Even in sequential mode TestNG could interleave test methods from the same suite. It does guarantee the sequence @BeforeClass -> @Test -> @AfterClass but it can do something like:

before class1
    test class1.method1
before class2
    test class2.method1
    test class1.method2
after class1
    test class2.method2
after class2

The solution is to force each class in a different suite (which are executed truly sequentially). As of version 2.16, the maven surefire plugin puts each class in a separate suite so the problem is fixed.

On the other hand, IDEA (even the latest 13 EAP) generates an xml file with all classes in the same suite. Hopefully IDEA will follow suit and fix this too. Interleaved tests are a showstopper when working with shared resources such as databases.

Answer 5

came here with a similar problem. I think group-by-instances="true" could be a solution.

Answer 6

TestNG offers several parallel strategies. It looks like methods is too aggressive for your needs but have you looked at classes or maybe instances?