问题
I have a Cucumber-JVM, JUnit, Selenium setup. I initiate the run by running RunSmokeTests.java using JUnit within Eclipse. I have also set up a maven profile to run the tests from command line, and possibly Jenkins in the future.
When the tests are run then some of them may fail sometimes, mainly due to the application taking longer than expected. I would then have to re-run these scenarios. At the moment I run them by manually attaching @rerun tag to the ones that failed and then running RunReruns.java, which is similar to RunSmokeTest.java but with @rerun tag.
With the increasing number of automated tests it is time consuming to tag the tests and start the run and clear the tags. Is there a automated way with Cucumber-JVM to re-run failed tests?
RunSmokeTests.java
package testGlueClasses;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@Cucumber.Options(features = "src/test/java", strict = true, format = {
"html:target/CucumberReport", "json:target/JSON/Cucumber.json",
"FrameworkCore.CustomTestReporter" }, tags = { "@SmokeTest" }, glue = {
"FrameworkCore", "MyApp.Utils", "MyApp.StepDefinitions" })
public class RunSmokeTests {
}
Maven snippet:
<profile>
<id>smoke</id>
<properties>
<include.tests>
**/RunSmokeTests.java
</include.tests>
</properties>
</profile>
回答1:
I came up with another solution to rerun just failed test using maven & cucumber.
1) Record test failures using a RunNotifier
public class RerunningCucumber extends Cucumber {
private final String className;
@SuppressWarnings("rawtypes")
public RerunningCucumber(Class clazz) throws InitializationError, IOException {
super(clazz);
className = clazz.getSimpleName();
}
@Override
public void run(RunNotifier notifier) {
notifier.addListener(new RunListener(){
public void testFailure(Failure failure) throws Exception {
Throwable error = failure.getException();
if (error instanceof AssertionError){
//Nothing. This is a normal failure. Continue
return;
}
//No! A wild exception has appeared!
//Let's run this test again.
RerunningCucumber.addFile(className);
}
});
super.run(notifier);
}
private static final String filename = "target/rerun.properties";
private static final Set<String> addedClasses = new HashSet<String>();
public static synchronized void addFile(String className) throws IOException{
//First find the file
if (addedClasses.contains(className)){
return;
}
File file = new File(filename);
if (!file.exists()){
//Need to create the file
PrintWriter writer = new PrintWriter(file, "UTF-8");
writer.print("retryclasses=**/"+className+".class");
writer.close();
}
else {
PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(file, true)));
out.print(",**/"+className+".class");
out.close();
}
addedClasses.add(className);
}
}
2) Use custom class as a runner for the cucumber tests.
This will run the tests, and whenever there is a failure, output the failed class to a file. Trick is to keep features short and create a lot of test classes to avoid repeating tests.
@RunWith(RerunningCucumber.class)
@CucumberOptions(features = {"classpath:features/testFeature.feature}, format = {
"html:target/cucumber-html-report/testFeature.html",
"json:target/cucumber-json-report/testFeature.json"},
tags = {"@testFeature"})
public class RunTestFeature {
}
3) Add a Rerun profile to maven.
This does three things: 1) it loads the failed classes into memory, 2) cleans JUST the failed classes properties file, and 3) reruns ONLY the failed tests as loaded from the properties file:
<profile>
<id>retry</id>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<version>1.0-alpha-2</version>
<executions>
<!-- Associate the read-project-properties goal with the initialize
phase, to read the properties file. -->
<execution>
<phase>pre-clean</phase>
<goals>
<goal>read-project-properties</goal>
</goals>
<configuration>
<files>
<file>target/rerun.properties</file>
</files>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-clean-plugin</artifactId>
<version>2.6.1</version>
<configuration>
<filesets>
<fileset>
<directory>target</directory>
<includes>
<include>rerun.properties</include>
</includes>
</fileset>
</filesets>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<version>1.6</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<echo>Retrying the following classes: "${retryclasses}"</echo>
</target>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.17</version>
<configuration>
<includes>
<include>${retryclasses}</include>
</includes>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
4) Usage
First test run:
mvn clean test
Next test runs:
mvn clean test -Pretry
mvn clean test -Pretry
mvn clean test -Pretry
...
You can repeat as many times as you want until there are no errors.
回答2:
I don't have an executable example at hand, but you can do this also on the jvm. There is a RerunFormatter that writes a text file listing the file and line numbers of failed scenarios:
@CucumberOptions(format = {"rerun:target/rerun.txt"})
You should be able to specify this file as input for another test class by prefixing it with @:
@CucumberOptions(features = {"@target/rerun.txt"})
回答3:
You can pass cucumber options to mvn as below
mvn clean verify -Dcucumber.options="@rerun.txt"
Note there is a tricky part here. If you are using the same test runner for both first run and rerun (and I believe that's what you want), then the test runner would contains something like
@CucumberOptions(plugin = { "rerun:target/rerun.txt"})
If you fire your rerun with maven using the same rerun file name as below
mvn clean verify -Dcucumber.options="@target/rerun.txt"
then cucumber will complain it could not find the rerun file. Why? Because the plugin "rerun:target/rerun.txt" will delete the file first with this test runner.
Workaround is copy/rename the file first, then kick off the mvn run like
mv target/rerun.txt rerun.txt && mvn clean verify -Dcucumber.options="@rerun.txt"
And this is actually what you want. Because say if there are 5 failed scenarios in file target/rerun.txt. And with the rerun after some fix, 2 of them passed. Now the target/rerun.txt will contain the remaining 3 failed scenarios only, which would be your new start point along the debugging way.
回答4:
You can use cucumber-jvm-parallel-plugin contributed code as a workaround until it goes live. Hit commands as shown below.
- git clone -b tagwiseOutlinewiseIssueRerun https://github.com/sugatmankar/cucumber-jvm-parallel-plugin.git
- mvn clean install.
- Now edit you project pom file and use as stated here.
- Example for using this plugin is here.
来源：https://stackoverflow.com/questions/21334207/rerunning-failed-cucumber-tests-using-cucumber-jvm