Qunit test alternates between pass and fail on page refresh

Question

I have a two tests that are causing side effects with each other. I understand why as I am replacing a jQuery built-in function that is being called internally in the second test. However what I don't understand is why the test alternately passes and fails.

This question is similar However, I am not doing anything directly on the qunit-fixture div.

Here are my tests

test('always passing test', function() { // Always passes
    var panelId = '#PanelMyTab';

    var event = {};
    var ui = {
        tab: {
            name: 'MyTab',
        },
        panel: panelId,
    };

    $('<div id="' + panelId + '">')
        .append('<a href="#" class="export">Test</a>')
        .append('<a href="#" class="showForm">Show Form</a>')
        .appendTo('#qunit-fixture');

    jQuery.fn.on = function(event, callback) {
        ok(this.selector == panelId + ' .export', 'Setting export click event');
        equal(callback, tickets.search.getReport, 'Callback being set');
    };

    loadTab(event, ui);
});

test('alternately passing and failing', function() { // Alternates between passing and failing on page refresh
    expect(5);

    var testUrl = 'test';
    $('<div class="ui-tabs-panel">')
        .append('<a href="'+ testUrl + '" id="getReport">Get Report</a>')
        .append('<form action="notest" target="" class="ticketSearch"></form>')
        .appendTo('#qunit-fixture');

    // Setup form mocking
    $('form.ticketSearch').submit(function() {
        var urlPattern = new RegExp(testUrl + '$');
        ok(urlPattern.test($(this).prop('action')), 'Form action set to link href');
        equal($(this).prop('target'), '_blank', 'Open form on a new page');
    });

    var event = {
        target: 'a#getReport',
    };

    var result = getReport(event);

    var form = $('form.ticketSearch');

    ok(/notest$/.test($(form).prop('action')), 'Making sure action is not replaced');
    equal($(form).prop('target'), '', 'Making sure that target is not replaced');

    ok(false === result, 'click event returns false to not refresh page');
});

The tests will start off passing but when I refresh they will alternate between passing and failing.

Why is this happening? Even adding GET parameters to the url result in the same behavior on the page.

In the failing cases, the test is failing because internal jQuery is calling .on() when the submit() handler is set. But why isn't the test always failing in that case? What is the browser doing that a state is being retained during page refresh?

Update:

Here is the code that is being tested:

var tickets = function() {
    var self = {
        loadTab: function(event, ui) {
            $(panel).find('.export').button().on('click', this.getReport);
        },

        search: {
            getReport: function(event) {
                var button = event.target;
                var form = $(button).closest('div.ui-tabs-panel').find('form.ticketSearch').clone(true);

                $(form).prop('action', $(button).prop('href'));
                $(form).prop('target', '_blank');

                $(form).submit();

                return false;
            }
        }
    };

    return self;
}();

Answer 1

I've modified @Ben's fiddle to include your code with both of your tests. I modified some of your code to make it run correctly. When you hit the run button all of the tests will pass. When you hit the run button again, the second test ("alternately passing and failing") will fail -- this is basically simulating your original issue.

The issue is your first test ("always passing test") alters the global state by replacing the jQuery.fn.on function with an overridden one. Because of this, when the tests are run in order, the second test ("alternately passing and failing") uses the incorrect overridden jQuery.fn.on function and fails. Each unit test should return the global state back to its pre-test state so that other tests can run based on the same assumptions.

The reason why it's alternating between pass and fail is that under the hood QUnit always runs failed tests first (it remembers this somehow via cookie or local storage, I'm not exactly sure). When it runs the failed tests first, the second test runs before the first one; as a result, the second test gets jQuery's native on function and works. When you run it a third time, the tests will run in their "original" order and the second test will use the overridden on function and fail.

Here's the working fiddle. I've add the fix to "un-override" the on function after the test by caching the original var jQueryOn = jQuery.fn.on; function and resetting it at the end of the test via: jQuery.fn.on = jQueryOn;. You can probably better implement this using QUnit's module teardown() method instead.

You can check out https://github.com/jquery/qunit/issues/74 for more info.

Answer 2

I'm not sure I can solve this without some more info, but I can point out some possible issues.

The first test seems to have invalid syntax on line 2

var panelId = '#PanelMyTab');

But that's probably a type mistake, seeing as you say the first always passes.

I'm assuming that for the first test to pass(and be valid) the loadTab(event,ui) must run the jQuery.fn.on(), without it no assertions have been run. Which doing some testing with jQuery UI Tabs, seems to be the case (just not sure if it was your intention).

I'm not sure it's advisable putting these assertions within that function, and you must understand that you have overwritten the jquery function with a function that doesn't do anything, so it's likely to cause issues.

You seem to be doing something similar in the second test, you are expecting 5 assertions, but I can only see how the final 3 can be run

ok(/notest$/.test($(form).prop('action')), 'Making sure action is not replaced');
equal($(form).prop('target'), '', 'Making sure that target is not replaced');
ok(false === result, 'click event returns false to not refresh page');

The other 2 are within a submit function that doesn't look like it is invoked as part of the test.

Remember these tests are synchronous so it won't wait for you to hit submit before running the test and failing.

Here is an example

test('asynchronous test', function() {
    setTimeout(function() {
        ok(true);
    }, 100)
})

Would fail as the ok is run 100ms after the test.

test('asynchronous test', function() {
    // Pause the test first
    stop();

    setTimeout(function() {
        ok(true);

        // After the assertion has been called,
        // continue the test
        start();
    }, 100)
})

The stop() tells qunit to wait and the start() to go!

There is also a asyncTest() detailed in the api here

Finally, it seems like you are trying to debug your code with these tests. It would be much easier to use chrome developer tools or firebug in firefox to set breakpoints on your code, and use console.log() and console.dir() to output information.

That being said I have no idea how it works for you at all, so I could be missing something :) If you're still stuck, see if you can add some more of the surrounding code and what your trying to achieve. Hope this helps.

PS: there is also a }; at the end which is invalid in the code you have given us, probably relevant in the actual application though ;)