Selenium + JUnit: test order/flow?

Question

I am using Selenium to test my java web app\'s html pages (JSPs actually). My web app requires a flow to access each pages (it is a small online game web app), as in: to get to

Answer 1

JUnit is not really designed for flow/integration level testing.

For that situation, I've designed my own Runner that guaranties the order the tests are run and also reuses the same instance of the test class for all test so you can pass values for a step to another.

(Yes, it's a bad practice for unit tests - but we're not talking about unit tests there even if they run with jUnit).

Using another tool (cucumber, fitnesse, TestNG, whatever) is also a good solution - but there is such thing as too many test tools in a project.

Answer 2

You can implement your own Runner that will probably wrap other Runner and sort tests according to criteria yo9u defined. This is if you really need this.

But I do not think it is so necessary. I understand that if page A does not work the pass to B will not work too. So you want to run the test A first and then run test A->B. But does it really make sense? If for example test A->B runs first and fails because it cannot arrive to page A other test that verifies test A will fail too. So, Both tests will fail and id does not depend on the tests order.

If however you mean that you want to use test A as a set up operation of test B it is very bad practice. You can use the logic of test A as a start of test B but you should not couple between 2 tests. One obvious reason is that this makes debugging very hard. To debug test A->B you have to run both A and A->B tests that means that you probably have to run all tests (at least within one test case).

Answer 3

Another option that may work is applying JUnit Parameterization to your test. It is my present understanding that the Parameters for the implementation are always executed in the order they are provided.

Using that concept you could have your JUnit implementation accept the URL as a constructor argument and fork the test internally based on the parameters provided.

In order to ensure that you're using the same WebDriver reference it would probably need to be static @BeforeClass/@AfterClass declarations. With that you may be able to have the parameters chain off one another, effectively testing that "From the previous test I am on page X. While here I will perform task Y. At the end of this test I will be on page Z, or in state A".

In Unit-level testing I would certainly say this solution would be bad form, but when you integrate a tool like Selenium you start acting on the integration-test level. I'm fairly new to this concept myself, but in the integration-test level it seems the rules of modularity are a bit more fuzzy since you will have conditions that are dependent.

I was curious, so I tried it out. It acts like I was thinking it would, if we assume we can treat the application as a static resource in relation to the test.

package demo.testing;

import java.util.List;

import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

import com.google.common.base.Predicate;
import com.google.common.base.Predicates;
import com.google.common.collect.Lists;

@RunWith(Parameterized.class)
public class SequentialParams {

    private static SystemState state;

    @BeforeClass
    public static void validateBeforeState() {
        state = new SystemState();

        Assert.assertFalse(state.one);
        Assert.assertFalse(state.two);
        Assert.assertFalse(state.three);
        Assert.assertFalse(state.four);
    }

    @Parameters
    public static Object buildParameters() {
        Runnable reset = new Runnable() {

            public void run() {
                state.one = false;
                state.two = false;
                state.three = false;
                state.four = false;
            }
        };
        Runnable oneToTrue = new Runnable() {

            public void run() {
                state.one = true;
            }
        };
        Runnable twoToTrue = new Runnable() {

            public void run() {
                state.two = true;
            }
        };
        Runnable threeToTrue = new Runnable() {

            public void run() {
                state.three = true;
            }
        };
        Runnable fourToTrue = new Runnable() {

            public void run() {
                state.four = true;

            }
        };


        Predicate<SystemState> oneIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.one;
            }
        };
        Predicate<SystemState> twoIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.two;
            }
        };
        Predicate<SystemState> threeIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.three;
            }
        };
        Predicate<SystemState> fourIsTrue = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return input.four;
            }
        };

        Predicate<SystemState> oneIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.one;
            }
        };
        Predicate<SystemState> twoIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.two;
            }
        };
        Predicate<SystemState> threeIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.three;
            }
        };
        Predicate<SystemState> fourIsFalse = new Predicate<SequentialParams.SystemState>() {
            public boolean apply(SystemState input) {
                return !input.four;
            }
        };
        List<Object[]> params = Lists.newArrayList();

        params.add(new Object[]{Predicates.and(oneIsFalse, twoIsFalse, threeIsFalse, fourIsFalse), oneToTrue, Predicates.and(oneIsTrue, twoIsFalse, threeIsFalse, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsFalse, threeIsFalse, fourIsFalse), twoToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsFalse, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsTrue, threeIsFalse, fourIsFalse), threeToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsFalse), fourToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsTrue)});

        params.add(new Object[]{ Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsTrue), reset, Predicates.and(oneIsFalse, twoIsFalse, threeIsFalse, fourIsFalse)});

        params.add(new Object[]{Predicates.and(oneIsFalse, twoIsFalse, threeIsFalse, fourIsFalse), threeToTrue, Predicates.and(oneIsFalse, twoIsFalse, threeIsTrue, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsFalse, twoIsFalse, threeIsTrue, fourIsFalse), oneToTrue, Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsFalse)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsFalse), fourToTrue, Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsTrue)});
        params.add(new Object[]{Predicates.and(oneIsTrue, twoIsFalse, threeIsTrue, fourIsTrue), twoToTrue, Predicates.and(oneIsTrue, twoIsTrue, threeIsTrue, fourIsTrue)});


        return params;
    }

    Predicate<SystemState> verifyStartState;
    Runnable changeState;
    Predicate<SystemState> verifyEndState;

    public SequentialParams(Predicate<SystemState> pre, Runnable task, Predicate<SystemState> post) {
      verifyStartState = pre;
      changeState = task;
      verifyEndState = post;
    }

    @Test
    public void perform() {
        Assert.assertTrue(verifyStartState.apply(state));
       changeState.run();
       Assert.assertTrue(verifyEndState.apply(state));
    }


    private static class SystemState {
        public boolean one = false;
        public boolean two = false;
        public boolean three = false;
        public boolean four = false;

    }

}

Answer 4

Why to migrate? You can use JUnit for unit-testing and another framework for higher-level testing. In your case it is a kind of acceptance or functional or end-to-end, it is not that important how you name it. But important is to understand that these tests are not unit. They stick to different rules: they are more complex, run longer and less often, they require complex setup, external dependencies and may sporadically fail. Why not use another framework for them (or even another programming language)?

Possible variants are:

• BDD frameworks: already mentioned Cucumber, JDave, JBehave, Spock. For instance, Spock is JUnit based and Groovy =)
• TestNG

If adding another framework is not an option: you enumerated more options for JUnit then I could imagine =) I would put the whole test script for the flow in one test method and would organize test code into "Drivers". That means that your end-to-end tests do not call the methods of your application or Selenium API directly, but wrap them into methods of Driver components which hide API complexity and look like statements of what happens or what is expected. Look at the example:

@Test 
public void sniperWinsAnAuctionByBiddingHigher() throws Exception {
    auction.startSellingItem();

    application.startBiddingIn(auction);
    auction.hasReceivedJoinRequestFrom(ApplicationRunner.SNIPER_XMPP_ID);

    auction.reportPrice(1000, 98, "other bidder");
    application.hasShownSniperIsBidding(auction, 1000, 1098);

    auction.hasReceivedBid(1098, ApplicationRunner.SNIPER_XMPP_ID);

    auction.reportPrice(1098, 97, ApplicationRunner.SNIPER_XMPP_ID);
    application.hasShownSniperIsWinning(auction, 1098);

    auction.announceClosed();
    application.hasShownSniperHasWonAuction(auction, 1098);
} 

A snippet is taken from the "Growing Object-Oriented Software Guided by Tests". The book is really great and I highly recommend to read it.

This is real end-to-end test that uses real XMPP connection, Openfire jabber server and WindowLicker Swing GUI-testing framework. But all this stuff if offloaded to Driver components. And in your test you just see how different actors communicate. And it is ordered: after application started bidding we check that auction server received join request, then we instruct auction server to report new price and check that it is reflected in UI and so on. The whole code is available on github.

The example on github is complex, because the application is not as trivial as it usually happens with book examples. But that book gives it gradually and I was able to built the whole application from scratch following the book guide. In fact, it is the sole book I ever read on TDD and automated developer testing that gives such a thorough and complete example. And I've read quite a lot of them. But note, that Driver approach does not make your tests unit. It just allows you hide complexity. And it can (and should) be used with other frameworks too. They just give you additional possibilities to split your tests into sequential steps if you need; to write a user readable test cases; to externalize test data into CSV,Excel tables, XML files or database, to timeout your tests; to integrate with external systems, servlet and DI containers; to define and run separately test groups; to give more user-friendly reports and so on.

And about making all your tests unit. It is not possible for anything excluding something like utility libraries for math, string processing and so on. If you have application that is completely unit tested that it means either that you test not all application or you do not understand what tests are unit and what are not. The first case may be OK, but everything that is not covered must be tested and retested manually by developers, testers, users or whoever. It is quite common but it better to be conscious decision rather than casual one. Why you cannot unit test everything?

There are a lot of definitions of unit tests and it leads to holy war) I prefer the following: "Unit test is test for program unit in isolation". Some people say: "hey, unit is my application! I test login and it is simple unit function". But there is also pragmatics that hides in isolation. Why do we need to differ unit tests from others? Because it is our first safety net. They must be fast. You commit often (to git, for example) and you run them at least before each commit. But imagine, that "unit" tests takes 5 minutes to run. You will either run them less often or you will commit less often or you will run just one test case or even one test method at the time, or you will wait say each 2 minutes for tests to complete in 5 minutes. An in that 5 minutes you'll go to Coding Horror where you'll spend the next 2 hours =) And unit tests must never fail sporadically. If they do that - you will not trust them. Hence, the isolation: you must isolate slowness and sources of sporadic failures from your unit tests. Hence, isolation means that unit tests should not use:

• File system
• Network, Sockets, RMI and so on
• GUI
• Multithreading
• External libraries expecting test framework and supporting simple libraries like Hamcrest

And unit tests must be local. You want to have just one or so tests failing when you've made a defect within 2 minutes of coding, not a half of the whole suite. That means that you are very limited in testing stateful behavior in unit tests. You should not make a test setup that makes 5 state transitions to reach preconditions. Because fail in first transition will break at least 4 tests for following transitions and one more test that you currently write for the 6th transition. And any non-trivial application has quite a lot of flows and state transitions in it. So this cannot be unit tested. For the same reason unit tests must not use changeable shared state in database, static fields, Spring context or whatever. This is exactly the reason why JUnit creates new instance of test class for every test method.

So, you see, you cannot fully unit test a web app, no matter how you recode it. Because it has flows, JSPs, servlet container and probably more. Of course, you can just ignore this definition, but it is damn useful) If you agree that distinguishing unit tests from other tests is useful, and this definition helps to achieve that then you'll go for another framework or at least another approach for tests that are not unit, you'll create separate suites for separate kinds of test and so on.

Hope, this will help)

Answer 5

We use a Framework call Cucumber. Its behavioral driven development. So essentially, you can create testing function that are agnostic of the flow and use feature to control the flow of your testing.

Example Page 1, enter 2 input, click enter, verify page 2 loaded with a something.

Then you'll have this in a feature file:

Given Page 1
   Then I enter text1
   Then I enter text2
   Then I click button
   Then I see page 2

And in you code, you can have a class that implements these steps. And with the cucumber framework, you can use annotation to denote the mapping between you testing code and the feature file.

...
@Given("^Page 1$")
public void iLoadPage1() {
  WebDriver driver = new ....
  driver.go('URL');
}

@Given("I enter (.*)$")
public void iEnterTest(String txt) {
  ...
}

...