Solid Java unit test automation? (JUnit/Hamcrest/…)

Question

Intent

I am looking for the following:

• A solid unit testing methodology
  1. What am I missing from my approach?
  2. What am I doing wro

Answer 1

1. Consider using ExpectedException instead of @Test(expected.... This is because if for example you expect a NullPointerException and your test throws this exception in your setup (before calling the method under test) your test will pass. With ExpectedException you put the expect immediately before the call to the method under test so there is no chance of this. Also, ExpectedException allows you to test the exception message which is helpful if you have two different IllegalArgumentExceptions that might be thrown and you need to check for the correct one.

2. Consider isolating your method under test from the setup and verify, this will ease in test review and maintenance. This is especially true when methods on the class under test are invoked as part of setup which can confuse which is the method under test. I use the following format:

   public void test() {
      //setup
      ...
   
      // test (usually only one line of code in this block)
      ...
   
      //verify
      ...
   }
   

3. Books to look at: Clean Code, JUnit In Action, Test Driven Development By Example

   Clean Code has an excellent section on testing

4. Most example I have seen (including what Eclipse autogenerates) have the method under test in the title of the test. This facilitates review and maintenance. For example: testOfComponents_nullCase. Your example is the first I have seen that uses the Enclosed to group methods by method under test, which is really nice. However, it adds some overhead as @Before and @After do not get shared between enclosed test classes.

5. I have not started using it, but Guava has a test library: guava-testlib. I have not had a chance to play with it but it seems to have some cool stuff. For example: NullPointerTest is quote:

• A test utility that verifies that your methods throw {@link * NullPointerException} or {@link UnsupportedOperationException} whenever any * of their parameters are null. To use it, you must first provide valid default * values for the parameter types used by the class.

Review: I realize the test above was just an example but since a constructive review might be helpful, here you go.

1. In testing getComponents, test the empty list case as well. Also, use IsIterableContainingInOrder.

2. In testing of ofComponents, it seems that it would make sense to call getComponents or toString to validate that it properly handled the various non-error cases. There should be a test where no argument is passed to ofComponents. I see that this is done with ofComponents( new String[]{}) but why not just do ofComponents()? Need a test where null is one of the values passed: ofComponents("blah", null, "blah2") since this will throw an NPE.

3. In testing ROOT, as has been pointed out before, I suggest calling ROOT.getComponents once and doing all three verifications on it. Also, ItIterableContainingInOrder does all three of not empty, size and contains. The is in the tests is extraineous (although it is linguistic) and I feel is not worth having (IMHO).

4. In testing toString, I feel it is very helpful to isolate the method under test. I would have written toStringIsSlashSeparatedPathOfComponents as follows. Notice that I do not use the constant from the class under test. This is because IMHO, ANY functional change to the class under test should cause the test to fail.

   @Test     
   public void toStringIsSlashSeparatedPathOfComponents() {       
      //setup 
      String[] components = { "Test1", "Test2", "Test3" };       
      String expectedPath =  "/" + Joiner.on("/").join(components);   
      MyPath path = MyPath.ofComponents(components)
   
      // test
      String value = path.toStrign();
   
      // verify
      assertThat(value, equalTo(expectedPath));   
   } 
   

5. Enclosed will not run any unit test that is not in an inner class. Therefore testPathCreationFromComponents would not be run.

Finally, use Test Driven Development. This will ensure that your tests are passing for the right reason and will fail as expected.

Answer 2

Well, I will post 2 difference answers.

1. As James Coplien stated unit test is worthless. I disagree with him on this issue, but may be you will find this helpful to consider to unit test less instead of searching for automatic solution.

2. Consider to use Theories with DataPoints. I think this will minimize your problem significantly. Also, using mock can help you.

Answer 3

I see you put a lot of effort to really test your classes. Good! :)

My comments/questions would be:

• what about mocking? you do not mention any tool for this
• it seems to me that you care a lot about the nitty-gritty details (I do not say they are not important!), while neglecting the business purpose of tested class. I guess it comes from the fact you code code-first (do you?). What I would suggest is more TDD/BDD approach and focus on business responsibilities of the tested class.
• not sure what this gives you: "Method testing grouped in static nested classes"?
• regarding auto-generation of test stubs etc. Simply put: don't. You will end up testing implementation instead of behaviour.

Answer 4

Ok, here is my view on your questions:

Is there a list of techniques to use to build a unit test?

Short answer, no. Your problem is that to generate a test for a method, you need to analyse what it does and put a test in for each possible value in each place. There are/were test generators, but IIRC, they didn't generate maintainable code (see Resources for Test Driven Development).

You've already got a fairly good list of things to check, to which I would add:

• Make sure all paths through your methods are covered.
• Make sure all important functionality is covered by more than one test, I use Parameterized a lot for this.

One thing I find really useful to do is to ask what should this method be doing, as opposed to what does this method do. This way, you write the tests with a more open mind.

Another thing I find useful is to cut down on the boilerplate associated with the tests, so I can read the tests more easily. The easier it is to add tests, the better. I find Parameterized very good for this. For me, readability of tests is key.

So, taking your example above, if we drop the requirement 'test only one thing in a method' we get

public static class Root {
  @Test
  public void testROOT() {
    assertThat("hasComponents", MyPath.ROOT.getComponents(), is(not(empty())));
    assertThat("hasExactlyOneComponent", MyPath.ROOT.getComponents(), hasSize(1));
    assertThat("hasExactlyOneInboxComponent", MyPath.ROOT.getComponents(), contains("ROOT"));
    assertThat("isNotNull", MyPath.ROOT, is(notNullValue()));
    assertThat("toStringIsSlashSeparatedAbsolutePathToInbox", MyPath.ROOT.toString(), is(equalTo("/ROOT")));
  }
}

I've done two things, I've added the description into the assert, and I've merged all of the tests into one. Now, we can read the test and see that we've actually got duplicate tests. We probably don't need to test is(not(empty()) && is(notNullValue()), etc. This violates the one assert per method rule, but I think it's justified because you've removed lots of boilerplate without cutting down on coverage.

Can I perform checks automatically?

Yes. But I wouldn't use annotations to do it. Let's say we have a method like:

public boolean validate(Foobar foobar) {
  return !foobar.getBar().length > 40;
} 

So I have a test method which says something like:

private Foobar getFoobar(int length) {
  Foobar foobar = new Foobar();
  foobar.setBar(StringUtils.rightPad("", length, "x")); // make string of length characters
  return foobar;
}

@Test
public void testFoobar() {
  assertEquals(true, getFoobar(39));
  assertEquals(true, getFoobar(40));
  assertEquals(false, getFoobar(41));
}

The above method is easy enough to factor out depending upon the length, into a Parameterized test of course. Moral of the story, you can factorize your tests just as you can with non-test code.

So to answer your question, in my experience, I've come to the conclusion that you can do a lot to help with all of the combinations by cutting down on boilerplate within your tests, by using a judicious combination of Parameterized and factorization of your tests. As a final example, this is how I would implement your test with Parameterized:

@RunWith(Parameterized.class) public static class OfComponents { @Parameters public static Collection data() { return Arrays.asList(new Object[][] { { new String[] {"Test1", "Test2", "Test3"}, null }, { new String[] {"Test1"}, null }, { null, NullPointerException.class }, { new String[] {"Test1", "", "Test2"}, IllegalArgumentException }, }); }

private String[] components;

@Rule
public TestRule expectedExceptionRule = ExpectedException.none();

public OfComponents(String[] components, Exception expectedException) {
   this.components = components;
   if (expectedException != null) {
     expectedExceptionRule.expect(expectedException);
   }
}

@Test
public void test() {
  MyPath.ofComponents(components);
}

Please note that the above isn't tested and probably doesn't compile. From the above, you can analyse the data as input and add (or at least think about adding) all of the combinations of everything. For instance, you haven't got a test for {"Test1", null, "Test2"} ...