In XCode 8, what is the way to make test methods run in a particular order within a given XCTestCase class?

Question

Prior to XCode 8, you could get your unit tests to run in a particular order by alphabetizing the names of the test methods within a given XCTestCase class (as described in this answer). E.g., tests would run like: testA, testB, testC, testD, etc.

However in XCode 8, this is no longer the case. For example I have test methods named test1, test2, test3, test4, and test4 will run first (see below screenshot). Then I can re-run, and test2 will run first on the next run-through.

So... how do I get the tests to run in order now on XCode 8...?

Answer 1

So the way I solved this is as follows.

Problem recap:

I needed to have several tests run in a row: test1, test2, test3, test4. Each test set up an expectation and the final step in the test would fulfill the expectation, then the test would conclude and the next one would run.

However in XCode 8, tests now run in random order. While that's good from the standpoint that, if they are unit tests, they should be able to run in random order, it breaks your tests if they are designed not as unit tests, but as end-to-end tests.

For example in my case it breaks the tests because in the first test, a user logs in and set up some data etc. Then, the second test checks the math, then the third test syncs the data to a server, then the fourth one deletes it all and syncs down from the server. When the first test runs, at build time, a shell script inits the server's DB from a MSYQL file, then at launch time of the app, the AppDelegate installs a fresh CoreData DB for the app. So, if I have to launch the app fresh after each test, the shell script will re-init the server DB and cause the app's local CoreData DB to also re-init. This will break the subsequent tests (it being an end-to-end test the subsequent tests depend on the state of the app and server being a certain way, after the previous test ran).

Rather than set up four different CoreData starting DBs and four different server init scripts (which would have been a huge pain and made the end-to-end test exponentially more time consuming to manage whenever we have a schema change), or have to remember to run without building each test manually in a row, instead I merged all four test methods into a single really long test method using the following tactic.

Solution

First, in the XCTestCase class, I set up an test expectation property:

@property (nonatomic, strong) XCTestExpectation *endOfTestExpectation;

At the end of the test1 in my XCTestCase class, I replaced its existing expectation with this new expectation, like so:

self.endOfTestExpectation = [self expectationWithDescription:
                                  @"endOfTestExpectation"];

[self waitForExpectationsWithTimeout:900 
                             handler:^(NSError * _Nullable error) {
    /* Code moved from test4's expectation completion block goes here */
}

For each test1 through test3, I moved the code that was inside the test's original expectation completion block into a new method called completion1 through completion3. For test4 I moved the code inside its original expectation completion block into the endOfTestExpectation's completion block at the end of the test1 method.

Then I renamed the methods test2 through test4 to be named t3st2 through t3st4 (quick and dirty, I know; you should pick something more descriptive after you get it working). At the end of the completion1 method, I call t3st2; at the end of completion2 I call t3st3; at the end of completion3 I call t3st4; and at the end of completion4 I call [self.endOfTestExpectation fulfill];.

This actually ends up being better than the old way, because in the old way, even if the first test failed, the subsequent tests would still run! Now, wherever an XCTFail happens, the entire thing just stops and we don't waste time running the rest if I was tabbed over into SO :D