Robot Framework: How to merge old/new statuses using rebot --merge and keep the only one?

问题

I'm using rebot tool to merge outputs after re-execution of failed tests.

robot --output original.xml /path/to/dir/.
robot --rerunfailed original.xml --output rerun.xml /path/to/dir/.
rebot -o machine1.xml -l machine1.html --merge original.xml rerun.xml

The same action is provided on few test machines. Test suite is identical and is executed against each VM. All VMs are considered as identical, however, they are not stable and I receive different results on each machine. I want to merge all results from all machines and retrieve the maximum number of Passed tests, to understand whether the test is REALLY failing or it's just unstable environment and test is OK by itself.

Other words, if test passed at least on 1 machine, but failed on other 3 machines, it needs to be considered as Passed in the final report.

However I receive False for such case.

Is it possible to change the behavior somehow?

Example from the final report:

Status: FAIL (critical)
Message:    Re-executed test has been merged.
New status: FAIL
New message: Re-executed test has been merged.
New status: FAIL
New message: IndexError: Given index 0 is out of the range 0--1.
Old status: FAIL
Old message: IndexError: Given index 0 is out of the range 0--1.
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message: 
Old status: PASS
Old message: Re-executed test has been merged.
New status: PASS
New message: 
Old status: PASS
Old message:

回答1:

Resolved by using simple script for prerebotmodifier written by myself. Maybe it will be helpful for somebody else.

So, I created TestStatusChecker.py which helps to merge reports with the following logic: test status will be failed ONLY if it was failing in ALL reports.

from robot.api import SuiteVisitor


class TestStatusChecker(SuiteVisitor):

    def __init__(self, *args):
        pass

    def visit_test(self, test):
        if 'PASS' in test.message and 'Re-executed test has been merged' in test.message:
            test.status = 'PASS'
            test.message = 'Test passed because it passed at least once.'

CLI command for merging the results:

rebot -l final_log.html --prerebotmodifier TestStatusChecker.py --merge 1.xml 2.xml 3.xml 4.xml report

来源：https://stackoverflow.com/questions/58661221/robot-framework-how-to-merge-old-new-statuses-using-rebot-merge-and-keep-the

标签

python

python-3.x

robotframework