How to make sharded protractor tests report after ALL tests finish (instead of per-file)?

Question

When sharding tests (aka running tests in parallel; ie: shardTestFiles: true), instead of reporting results when all tests are finished, Protractor rep

Answer 1

Here is another solution, building on top of this answer, using protractor-html-reporter-2 which works on the xml file produced by jasmine-reporters. But jasmine-reporters does not have any option to work with reports generated by multiple instances of browsers. After failing to find an ideal solution, I ended up doing following in the protractor config js file:

// add relevant packages in package.json
'use strict';
const HTMLReport = require('protractor-html-reporter-2');
const jasmineReporters = require('jasmine-reporters');
const moment = require('moment');
const os = require('os');
const xmldoc = require('xmldoc');
...
const DATE_FORMAT = 'YYYYMMDD-HHmmss-SSS'; // use any other format that gives unique timestamp
const reportDir = path.join(__dirname, '../report');
...
exports.config = {
    ...
    framework: 'jasmine',
    capabilities: {
        browserName: 'chrome',
        maxInstances: 2,
        shardTestFiles: true,
    },
    beforeLaunch: async function () {
        // clean up report directory
        fs.emptyDirSync(reportDir);
    }, 
    onPrepare: async function () {
         const NOW = moment().format(DATE_FORMAT);
         const reportName = 'index-' + NOW;
         jasmine.getEnv().addReporter(new jasmineReporters.JUnitXmlReporter({
             consolidateAll: true,
             savePath: reportDir,
             filePrefix: reportName,
         }));
    },
    onComplete: async function () {
        // do something after each instance of browser is closed
    },
    afterLaunch: async function (exitCode) {
        // do something after ALL instances of browser are closed
        await consolidateJasmineXmlReports();
    },
    ...     
},
...
async function consolidateJasmineXmlReports() {
    // there may be better ways to write xml out but this works for me
    const files = fs.readdirSync(reportDir).filter(fn => fn.endsWith('.xml'));
    let disabledSum = 0;
    let errorsSum = 0;
    let failuresSum = 0;
    let testsSum = 0;
    let timeSum = 0;
    const allTestSuiteNodes = [];
    for (const file of files) {
        const pathToXml = reportDir + path.sep + file;
        console.log('Reading xml report file: ' + pathToXml);
        const xml = fs.readFileSync(pathToXml);
        const xmlDoc = new xmldoc.XmlDocument(xml);
        const disabled = parseInt(xmlDoc.attr.disabled);
        const errors = parseInt(xmlDoc.attr.errors);
        const failures = parseInt(xmlDoc.attr.failures);
        const tests = parseInt(xmlDoc.attr.tests);
        const time = parseFloat(xmlDoc.attr.time);
        disabledSum += disabled;
        errorsSum += errors;
        failuresSum += failures;
        testsSum += tests;
        timeSum += time;

        const testSuiteNodes = xmlDoc.childrenNamed('testsuite');
        allTestSuiteNodes.push(testSuiteNodes);
    }

    let startXml = `<?xml version="1.0" encoding="UTF-8" ?>`;
    startXml += `<testsuites disabled="` + disabledSum + `" errors="` + errorsSum + `" failures="` + failuresSum + `" tests="` + testsSum + `" time="` + timeSum + `">`;
    const endXml = '</testsuites>';
    allTestSuiteNodes.push(endXml);
    const finalXml = startXml + allTestSuiteNodes.join('\n');
    fs.writeFileSync(reportDir + path.sep + 'consolidated.xml', finalXml, 'utf8');
        
    const testConfig = {            
        outputPath: reportDir,
        outputFilename: 'consolidated',
        ...
    };

    new HTMLReport().from(reportDir + path.sep + 'consolidated.xml', testConfig);
}

The logic is

1. Ensure unique names for all xml files.
2. Combine all the xml files to one valid xml in the afterLaunch.
3. Use any package that uses that xml file to produce html in the afterLaunch.

We use Jenkins to run the tests and the report created by above displays well in Jenkins and also shows up accurately on the report displayed by Open Blue Ocean Jenkins plugin.

Note: I have tested with shardTestFiles and not with multiCapabilities but I think it should work with that as well.

See Also:

• Protractor config file
• Setting up jasmine reporter

Answer 2

Solution2-> Step 1. From the latest version of html-reporter, filePrefix should be fileNamePrefix, so function should be:

onPrepare: function() {
        return new Promise(function (fulfill, reject) {
            browser.getCapabilities().then(function (value) {
                reportName = value.get('webdriver.remote.sessionid') + '_' + value.get('browserName') + '_' + Math.floor(Math.random()*1E16);
                jasmine.getEnv().addReporter(
                    new Jasmine2HtmlReporter({
                        savePath: 'target/',
                        screenshotsFolder: 'images',
                        consolidate: true,
                        consolidateAll: true,
                        fileNamePrefix: reportName + ".html"
                    })
                );
                fulfill();
            })
        });
    },

Answer 3

I am afraid there is not a easy answer to this,as Protractor overwrites the report files when using any custom plugins. But the below two worked for me. Choose what suits you best

1) Tinker with 'index.js' of Jasmine2HtmlReporter to append file instead of PhantomJs overwrite its using

2) Generate unique HTML reports by configuring Jasmine2HTML reporter from onPrepare() function and consolidate all the reports later

SOLUTION 1: The current code base of Jasmine2HtmlReporter - index.js uses two functions - phantomWrite() & nodeWrite() to write data. Refer here

I have created a new function - appendwrite() to append instead of overwriting and have modified code to pickup this function Check out my github code forked out of protractor-jasmine2-html-reporter

function appendwrite(path, filename, text){ var fs = require("fs"); var nodejs_path = require("path"); require("mkdirp").sync(path); // make sure the path exists var filepath = nodejs_path.join(path, filename); fs.appendFileSync(filepath,text) return; }

And modify the self.writeFile function in 'node_modules/protractor-jasmine2-html-reporter/index.js' to pickup the new function try { appendwrite(path, filename, text); //phantomWrite(path, filename, text); return; } catch (e) { errors.push(' PhantomJs attempt: ' + e.message); } try { nodeWrite(path, filename, text); return; } catch (f) { errors.push(' NodeJS attempt: ' + f.message); }

And Comment the below code which cleans reports on new run so that you dont see any error cleanup error - CleanUpCode

    rmdir(self.savePath);

SOLUTION 2: Generate separate reports based on sessionID for parallel instances by configuring the Jasmine reporter in OnPrepare function

onPrepare: function() { return new Promise(function (fulfill, reject) { browser.getCapabilities().then(function (value) { reportName = value.get('webdriver.remote.sessionid') + '_' + value.get('browserName') + '_' + Math.floor(Math.random()*1E16); jasmine.getEnv().addReporter( new Jasmine2HtmlReporter({ savePath: 'target/', screenshotsFolder: 'images', consolidate: true, consolidateAll: true, filePrefix: reportName + ".html" }) ); fulfill(); }) }); },

Step 2: Consolidate the reports generated across parallel instances in afterLaunch() method after complete tests are done and all webdriver sessions are closed

afterLaunch: function afterLaunch() { var fs = require('fs'); var output = ''; fs.readdirSync('target/').forEach(function(file){ if(!(fs.lstatSync('target/' + file).isDirectory())) output = output + fs.readFileSync('target/' + file); }); fs.writeFileSync('target/ConsolidatedReport.html', output, 'utf8'); },

You will see reports generated something like below with one ConsolidatedReport also PS: Please ignore any typo and syntax errors. this is just to serve as an example and can be customized

I have answers this earlier at protractor-jasmine2-html-reporter doesn't consolidate results for all test when tests are shared using 'shardTestFiles': true in conf file