Automatically create test cases for web page?

Question

If someone has a webpage, the usual way of testing the web site for user interaction bugs is to create each test case by hand and use selenium.

Is there a tool to create

Answer 1

I think Growing Test Cases Automatically is more of what your asking. To be more specific I'll try to introduce basics and if you're interested take a closer look at Evolutionary Testing

Usually there is a standard set of constraints we meet like changing functionality of the system under test (SUT), limited timeframe, lack of appropriate test tools and the list goes on… Yet there is another type of challenge which arises as technological solutions progress further – increase of system complexity.

While the typical constraints are solvable through different technical and management approaches, in the case of system complexity we are facing the limit of our capability of defining a straight-forward analytical method for assessing and validating system behavior. Complex system consist of multiple, often heterogeneous components which when working together amplify each other’s statistical and behavioral deviations, resulting in a system which acts in ways that were not part of its initial design. To make matter worse, complex systems increase sensitivity to their environment as well with the help of the same mechanism.

Options for testing complex systems

How can we test a system which behaves differently each time we run a test scenario? How can we reproduce a problem which costs days and millions to recover from, but happens only from time to time under conditions which are known just approximately?

One possible solution which I want to focus on is to embrace our lack of knowledge and work with whatever we have by using evolutionary testing. In this context the evolutionary testing can be viewed as a variant of black-box testing, because we are working with feeding input into and evaluating output from a SUT without focusing on its internal structure. The fine line here is that we are organizing this process of automatic test case generation and execution on a massive scale as an iterative optimization process which mimics the natural evolution.

Evolutionary testing

Elements:

• Population – set of test case executions, participating into the optimization process

• Generation – the part of the Population, involved into given iteration

• Individual – single test case execution and its results, an element from the Population

• Genome – unified definition of all test cases, model describing the Population

• Genotype – a single test case instance, a model describing an Individual, instance of the Genome

• Recombination – transformation of one or more Genotypes into a new Genotype

• Mutation – random change in a Genotype

• Fitness Function – formalized criterion, expressing the suitability of the Individual against the goal of the optimization

How we create these elements?

• Definition of the experiment goal (selection criteria) – sets the direction of the optimization process and is related to the behavior of the SUT. Involves certain characteristics of SUT state or environment during the performed test case experiments. Examples: o “SUT should complete the test case execution with an error code” o “The test case should drive the SUT through the largest number of branches in SUT’s logical structure” o “Ambient temperature in the room where SUT is situated should not exceed 40 ºC during test case execution” o “CPU utilization on the system, where SUT runs should exceed 80% during test case execution” Any measurable parameters of SUT and its environment could be used in a goal statement. Knowledge of the relation between the test input and the goal itself is not obligatory. This gives a possibility to cover goals which are derived directly from requirements, rather than based on some late requirement derivative like business, architectural or technical model. • Definition of the relevant inputs and outputs of the tested system – identification of SUT inputs and outputs, as well as environment parameters, relevant to the experiment goal. • Formal definition of the experiment genome – encoding the summarized set of test cases into a parameterized model (usually a data structure), expressing relevant SUT input data, environment parameters and action sequences. This definition also needs to comply with the two major operations applied over genome instances – recombination and mutation. The mechanism for those two operations can be predefined for the type of data or action present in the genome or have custom definitions • Formal definition of the selection criteria (fitness function) – an evaluation mechanism which takes SUT output or environment parameters resulting from a test case execution (Individual) and calculates a number (Fitness), signifying how close is this particular Individual to the experiment goal.

How the process works?

1. We use the Genome to create a Generation of random Genotypes (test case instances).
2. We execute the test cases (Genotypes) generating results (Individuals)
3. We evaluate each execution result (Individual) against our goal using the Fitness Function
4. We select only those Individuals from given Generation which have Fitness above a given threshold (the top 10 %, above the average, etc.)
5. We use the selected individuals to produce a new, full Generation set by applying Recombination and Mutation
6. We repeat the process, returning on step 2 The iteration process usually stops by setting a condition with regard to the evaluated Fitness of a Generation. For example: • If the top Fitness hasn’t changed with more than 0.1% since the last Iteration • If the difference between the top and the bottom Fitness in a Generation is less than 0.3% then probably it is time to stop.

Upsides and downsides

Upsides:

• We can work with limited knowledge for the SUT and goal-oriented test definitions

• We use a test case model (Genome) which allows us to mass-produce a large number of test cases (Genotypes) with little effort

• We can “seed” test cases (Genotypes) in the first iteration instead of generating them at random in order to speed up the optimization process.

• We could run test cases in parallel in order to speed up the process

• We could find multiple solutions which meet our test goal

• If the optimization process in convergent we have a guarantee that each following Generation is a better approximate solution of our test goal. This means that even if we need to stop before we have reached optimal Fitness we will still have better test cases than the one we started with.

• We can achieve replay of very complex, hard to reproduce test scenarios which mimic real life and which are far beyond the reach of any other automated or manual testing technique.

Downsides:

• The process of defining the necessary elements for evolutionary test implementation is non-trivial and requires specific knowledge.

• Implementing such automation approach is time- and resource-consuming and should be employed only when it is justifiable.

• The convergence of the optimization process depends on the smoothness of the Fitness Function. If its definition results in a zones of discontinuity or small/no gradient then we can expect slow or no convergence

Update:

I also recommend you to look at Genetic algorithms and this article about Test data generation can give you approaches and guidelines.