Browserstack reports successful even when test fails in Nightwatchjs

Question

I just started using nightwatch with browserstack and I'm noticing that when we get a failed test, nightwatch registers the failure, but browserstack does not. sample test I am using. Also I am using free trial version of BrowserStack.

My question is:

1. Are there any ideas how to tell browserstack when a test run failed ?

From BrowserStack doc:

REST API

It is possible to mark tests as either a pass or a fail, using the following snippet:

var request = require("request"); 
request({
    uri: "https://user:key@www.browserstack.com/automate/sessions/<session-id>.json",
    method: "PUT",
    form: {
        "status": "completed",
        "reason":""
    }
});

The two potential values for status can either be completed or error. Optionally, a reason can also be passed.

My questions are:

2. How I can get 'session-id' after test execution ?
3. What if I can see "completed" status in dashboard already ?

Answer 1

1. A session on BrowserStack has only three types of statuses: Completed, Error or Timeout. Selenium (and hence, BrowserStack) does not have a way of understanding, if a test has passed or failed. Its by the multiple assertions in your tests that appear on your console, that you infer if a test has passed / failed. These assertions however, do not reach BrowserStack. As you rightly identified, you can use the REST-API, to change the status of the session to 'Error', if you see a failure in your console.

2. I would suggest fetching the session ID of the test as the test is being executed, since fetching the session ID after the test execution is a lengthy process. In Nightwatch, you can fetch session ID as follows:

browser.session(function(session) {
    console.log(session.sessionId);
});

3. Yes, you can certainly change the status of the session once it is completed. That's where the REST-API comes to help!

Answer 2

If you came here searching for a solution in Python, you could use

requests.put(
    "https://api.browserstack.com/automate/sessions/{}.json".format(driver.session_id),
    auth=(USERNAME, ACCESS_KEY),
    json={"status": "failed", "reason": "test failed"})