cucumber.json report getting overwritten by rerun scenario report

Question

I have got UI Test project and a API test project with same technology stack (JAVA1.8, Cucumber-JVM, JUnit, Maven) and both projects are showing me this problem. Probably because same set of dependencies are present in both.

I have employed the Flaky test re-run mechanism using maven-surefire-plugin build-in functionality <rerunFailingTestsCount>1</rerunFailingTestsCount>. Also, I have cucumber dependencies added based on <groupId>io.cucumber</groupId> and not <groupId>info.cukes</groupId>. Both these have their own version of cucumber-java and cucumber-jvm dependencies.

My POM.XML looks like this.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.21.0</version>
    <configuration>
         <rerunFailingTestsCount>1</rerunFailingTestsCount>
    </configuration>
    <executions>
         <execution>
             <id>acceptance-test</id>
             <phase>integration-test</phase>
             <goals>
                 <goal>test</goal>
             </goals>
             <configuration>
                 <forkCount>1</forkCount>
                 <reuseForks>true</reuseForks>
                  <includes>
                      <include>**/*Runner.class</include>
                  </includes>
              </configuration>
         </execution>
    </executions>
</plugin>
<dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-java</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-junit</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-spring</artifactId>
        <version>2.4.0</version>
    </dependency>
    <dependency>
        <groupId>io.cucumber</groupId>
        <artifactId>cucumber-jvm</artifactId>
        <version>2.4.0</version>
        <type>pom</type>
    </dependency>

ONLY RUNNER FILE CODE

@RunWith(Cucumber.class)
@ContextConfiguration(locations = {"file:/src/test/resources/spring-config.xml"})
@CucumberOptions(
        glue = "com.test.uitest",
        features = "classpath:cucumber",
        tags = {"~@ignore","@ui_home"},
        monochrome = true,
        plugin = {"pretty", "html:target/cucumber-reports",
                "json:target/cucumber-reports/cucumber.json",
                "rerun:target/rerun.txt"} //Creates a text file with failed scenarios
)
public class AllTestsRunner {
}

Now Aparently, I need to have another runner with following code in it (as per other forums and threads here on StackOverflow)

@RunWith(Cucumber.class)
@CucumberOptions(
        monochrome = true,
        glue = "com.test.uitest",
        features = "@target/rerun.txt", //Cucumber picks the failed scenarios from this file
        format = {"pretty", "html:target/rerun-reports",
                "json:target/cucumber-reports/rerun-report.json"}
)
public class FailedTestsRunner {
}

But I don't need to have this 2nd runner, as the rerun mechanism works absolutely brilliantly with just the one runner on top. Even, there is no need of rerun.txt file generated in 1st runner.The in-build mechanism within maven-surefire plugin (v_2.21.0) along with io.cucumber v_2.4.0 works perfectly and if any scenario fails during 1st execution, it reruns automatically without recording it inside rerun.txt file.

## PROBLEM IS ##

I have 5 scenarios in my feature file. If all of them pass in 1st run. It successfully generates the report with cucumber.json report showing all 5 scenarios. But, if (say) 2 out of 5 scenarios fails, and those gets executed automatically in a rerun mechanism, the cucumber.json report file only recording results of those two scenarios and not the all 5 scenarios. Overall build PASSES if those 2 scenarios passes in rerun or FAILs if those 2 scenarios fails. That's correct, but my problem is with the cucumber.json getting overwritten by rerun mechanism. I have tried to use the maven-cucumber-reporting plugin v_3.16.0 but it actually reads the cucumber.json file itself and hence don't resolve my problem. Any help would be appreciated.

Answer 1

The good news is you're not doing anything wrong!

The bad news is that the results you're observing are entirely as expected. This is a natural consequence of chaining different tools (Surefire --> JUnit --> Cucumber) that are otherwise unaware of each other. From the perspective of Cucumber it would appear that the rerun is an entirely new execution so it will happily overwrite the old reports. Only at the start of the chain it is possible to create accurate reports.

As such your options from least to most effort and worst to best quality are:

1. Use the reports generated by Surefire.

2. Write your own plugin that appends results rather then overwriting them.

3. Get involved with the Cucumber project and help resolve this fundamentally.

Edit: Removed the suggestion to use the cucumber-jvm-parallel-plugin create a individual unit test for each scenario. This won't actually work.

Answer 2

In this case the AllTestsRunner is run repeatedly till it succeeds up to the count mentioned in the rerunFailingTestsCount parameter or fails on last run. So the last run always wins.

The surefire plugin creates its own reports in the target folder. Two reports are AllTestsRunner.txt, which is a summary, and TEST-AllTestsRunner.xml has the details. Both these reports has details about all runs. You could create a custom program to convert the TEST-AllTestsRunner.xml file to desired json.

There is a surefire report plugin which reads the above xml file and generates html report. It will create reports in the site folder and run with mvn site. Maybe this works.