Parameterized unit test suites

Question

I am trying to set up some parameterized test suites, unfortunately without any luck so far. I have two set of parameters, and I would like to run multiple test cases (they are in different classes) with all possible combinations. I tried to do it with JUnit4, but I am unable to set it up correctly. This would be my basic idea:

1. TestSuite1.class sets up one set of parameters, then it starts TestSuite2.class.
2. TestSuite2.class sets up the second set of parameters, then it starts the actual test(s) that will use both parameters.

Meanwhile it seems it is not possible to set up both Suite.class and Parameterized.class in the RunWith annotation at the same time (according to google, Parameterized extends Suite, I get usually "no runnable method found" message if I use.)

This is how my code looks like basically:

TestSuite1.class:

@RunWith(Parameterized.class)
@Parameterized.SuiteClasses({TestSuite2.class})
//I have tried with @RunWith(Suite.class) and
//@Suite.SuiteClasses({TestSuite2.class}) annotations also - all combinations
public class TestSuite1{

  public TestSuite1(int number) {
    Params.first = number;
  } 

  @Parameters
  public static Collection<Object[]> parameters(){
    Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
    return Arrays.asList(data);
  }
}

TestSuite2.class looks the same as TestSuite1.class, except that I have added TestCase1.class to the suite instead of TestSuite2, and that it sets another variable in Params.

TestCase1.class:

public class TestCase1 {    
  @Test
  public void test1(){
    System.out.println("first: "+Params.first+" second: "+Params.second);
    Assert.assertTrue(true);
  }
}

I am open to all ideas - even with TestNG for example. I have tried it also (although today was the first time I saw it), but as I noticed the suites are a bit different than in JUnit. I would prefer not to set up XML files before testing, I would like to solve all set up programmatically.

Is what I am trying to achieve possible with any framework?

---

Update: With TestNG I have the following code:

Start.class:

public class Start {

public static void main(String[] args){
    TestListenerAdapter tla = new TestListenerAdapter();
    TestNG testng = new TestNG();
    testng.setTestClasses(new Class[] { FirstTest.class, SecondTest.class });
    testng.addListener(tla);
    testng.run();
}
}

Params.class:

public class Params {
@DataProvider(name = "param")
public static Object[][] createData() {
    Object[][] data = new Object[][] { { 1 }, { 2}, { 3}, { 4} };
    return data;
  }
}

FirstTest.class:

public class FirstTest {

@Test(dataProvider = "param", dataProviderClass = Params.class)
public static void printIt(int number){
    System.out.println("FirstTest: "+number);
}

}

SecondTest.class is the same as FirstTest.class. If I run this, it runs FirstTest 4 times, then it runs SecondTest 4 times. I would like to run FirstTest one time, and SecondTest one time also with the first set of parameters. Then I would like to run FirstTest and SecondTest one time, with the second set of parameters, etc.

I have tried to set setPreserveOrder(true), and tried all setParallel options also. On this way however the results are in kind of random order.

(It would be some selenium test. I am aware that tests should not depend on each other, but still it would be my desired way for this)

Answer 1

So basicly as far as I understand, what you want to do is to run a test with set of sets of parameters. That is possible with JUnit and that is why method annotated with @Parameters returns a Collection of arrays (in general a set of sets).

Look at this example:

import static org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class TestCase1 {

    public TestCase1(int first, int second) {
        Params.first = first;
        Params.second = second;
    }

    @Parameters
    public static Collection<Object[]> parameters(){
        Object[][] data = new Object[][] { { 1, 11 }, { 2, 22 }, { 3, 33 }, { 4, 44 } };
        return Arrays.asList(data);
    }

    @Test
    public void test1(){
        System.out.println("first: "+Params.first+" second: "+Params.second);
        Assert.assertTrue(true);
    }
}

Edit: If you want to share parameters between multiple test you could use an abstraction in your test cases.

public class AbstractParametrizedTest {

    public AbstractParametrizedTest(int first, int second) {
        Params.first = first;
        Params.second = second;
    }

    @Parameterized.Parameters
    public static Collection<Object[]> parameters(){
        Object[][] data = new Object[][] { { 1, 11 }, { 2, 22 }, { 3, 33 }, { 4, 44 } };
        return Arrays.asList(data);
    }
}


@RunWith(Parameterized.class)
public class TestCase1 extends AbstractParametrizedTest {

    public TestCase1(int first, int second) {
        super(first, second);
    }

    ...
}

However the best way to do it i think, would be to use TestNGs data providers. Take a look at example at section 5.6.2 and usage of static data providers http://testng.org/doc/documentation-main.html

Answer 2

To achieve the goal of executing all test cases with the same parameters in sequential order, you would need a different Runner as this behavior is held in that class. You're in luck, as this is available in the JUnit Toolbox Project with the ParallelParameterized class!

Answer 3

Although Parameterized extends Suite, it behaves totally different - in disrespect of the Liskov substitution principle. This is because normally the constructor Suite(Class<?>, RunnerBuilder) processes the @SuiteClasses annotation. But Parameterized(Class<?>) replaces this behaviour with a processing of @Parameters.

If you want to combine the behaviour of Suite and Parameterized you have to look outside of JUnit 4. E.g. you could implement your own Runner like Adam Hawkes already mentioned in another post here.

I did the same by myself and cobbled a library together that provides you with a ParameterizedSuite Runner: https://github.com/PeterWippermann/parameterized-suite

A parameterized test suite looks like this:

@RunWith(ParameterizedSuite.class)
@SuiteClasses({OneTest.class, TwoTest.class})
public class MyParameterizedTestSuite {
    @Parameters(name = "Parameters are {0} and {1}")
    public static Object[] params() {
        return new Object[][] {{'A',1}, {'B',2}, {'C',3}};
    }

Answer 4

Here some other suggest that seems to be much more flexible: @RunWith(Enclosed.class)

In short: Instead of @Suite.SuiteClasses(...), just use @RunWith(Enclosed.class) and extend your Test Classes

@RunWith(Enclosed.class)
public class FastTest {

    public static class Test1FirstAppInit extends AppInitTest {    }

    public static class Test2Download extends DownloadTest{    }

    public static class Test3OtherTest extends OtherTest {    }
}

Now with Parameterized:

@RunWith(Enclosed.class)
public class FastTest {

    private static Iterable<? extends Object> mAllLocale = Arrays.asList(Locale.ENGLISH, Locale.GERMAN);
    private static Iterable<? extends Object> mSingleLocale = Arrays.asList(Locale.ENGLISH);

    /*
    Run test class for all Locale
     */
    @RunWith(Parameterized.class)
    public static class Test1FirstAppInit extends AppInitTest {
        @Parameterized.Parameter
        public Locale mLocale;

        @Parameterized.Parameters
        public static Iterable<? extends Object> data() {
            return mAllLocale;
        }

        @Override
        public Locale getLocale() {
            return mLocale;
        }

        @Override
        public void test001ResetAll {
            assumeTrue(false); // skip test completedly
        }


        @Override
        public void test002ClearAppData() {
            // replace existing test
            if (getLocale() != Locale.ENGLISH) {
                /*
                should run only on first Locale
                skip test on following Parameter runs
                 */
                assumeTrue(false); // skip test
            }
            else {
                super.test000ClearAppData();
            }
        }
    }

    /*
    Run test class only for one Locale
     */
    @RunWith(Parameterized.class)
    public static class Test2Download extends DownloadTest{
        @Parameterized.Parameter
        public Locale mLocale;

        @Parameterized.Parameters
        public static  Iterable<? extends Object> data(){
            return mSingleLocale;
        }

        @Override
        public Locale getLocale() {
            return mLocale;
        }

        @Override
        public void test900Delete() {
            assumeTrue(false); // skip test
        }
    }

    /*
    Test not Parameterized
     */
    public static class Test3OtherTest extends OtherTest {    }
}

Your Test Classes for Parameterized tests look like this:

@RunWith(AndroidJUnit4.class)
@LargeTest
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class DownloadTest {

    public Locale getLocale() {
        // will be overwritten in @RunWith(Enclosed.class)
        // but we are still able to run test class separatedly
        return Locale.ENGLISH;
    }

    @Test
    public void test900Delete() {
        ....
    }
    ....
}

Matches exactly what I was searching for. I can create different Test scenarios (full test, fast test,...). Just create different @RunWith(Enclosed.class) classes and extend the tests that you want to include.

Only side point seems to be that Enclosed.class does not care about sort order (if important to you). I solved it by replacing Enclosed:

public class SortedEnclosed extends Suite {

    public SortedEnclosed(Class<?> klass, RunnerBuilder builder) throws Throwable {
        super(builder, klass, filterAbstractClasses(klass.getClasses()));
    }

    protected static Class<?>[] filterAbstractClasses(final Class<?>[] classes) {
        final List<Class<?>> filteredList= new ArrayList<Class<?>>(classes.length);

        for (final Class<?> clazz : classes) {
            if (!Modifier.isAbstract(clazz.getModifiers())) {
                filteredList.add(clazz);
            }
        }
        // this is new (there may be better way with own "@FixClassOrder"...):
        Collections.sort(filteredList, new Comparator<Class<?>>() {
            @Override
            public int compare(Class<?> o1, Class<?> o2) {
                return o1.getSimpleName().compareTo(o2.getSimpleName());
            }
        });
        //  
        return filteredList.toArray(new Class<?>[filteredList.size()]);
    }
}

And then use @RunWith(SortedEnclosed.class)