Can I run an XCTest suite multiple times?

Question

Is it possible to have Xcode run your unit tests multiple times?

I had an issue in several unit tests that caused intermittent failures. Now that I think I\

Answer 1

I had used the invokeTest() override in the past (Xcode 10) with great success. But now in Xcode 11 its not working (for me at least). What I ended up doing was:

func test99Loop() {
    for i in 0..<LOOP_COUNT {
        if i > 0 { tearDown(); sleep(1); setUp() }
        test3NineUrls()
        do { tearDown(); sleep(1) }
        setUp()
        test6NineCombine()

        print("Finished Loop \(i)")
    }
}

I obviously use setup/teardown, and this is the proper way to do those multiple times (since the first and last are called by Xcode).

Answer 2

Try overriding invoke test: https://developer.apple.com/documentation/xctest/xctestcase/1496282-invoketest?language=objc

- (void)invokeTest
{
    for (int i=0; i<100; i++) {
        [super invokeTest];
    }
}

Answer 3

It might help you to use

func testMultiple() {
    self.measureBlock() {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

This runs the code inside self.measureBlock() multiple times to measure the average time.

It is work to change the code, but you might want to know the execution time anyways.

This answer might be close enough to what you want and it is easy to do.

Answer 4

Try using a for loop:

func testMultiple() {
    for _ in 0...100 {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

Note this doesn't work within a self.measureBlock(). You'll get an NSInternalConsistencyException: Cannot measure metrics while already measuring metrics

However, you can CALL this within a measureBlock():

func testMultiple() {
    for _ in 0...100 {
            ...
            XCTAssert(errMessage == nil, "no error expected")        
    }
}

func testPerformance() {
    self.measureBlock() {
        self.testMultiple()
    }
}

Xcode 8 runs the measureBlock code 10 times.

Answer 5

One alternative is to do this via the command line. You can run a single test using the -only-testing argument, and avoid building using test-without-building i.e. (new lines added for clarity)

for i in {1..10}; \
  do xcodebuild \
    test-without-building \
    -workspace MyApp.xcworkspace \
    -scheme Debug \
    -destination 'platform=iOS Simulator,OS=11.2,name=iPhone 8' \
    -only-testing:MyApp.Tests/TestFile/myTest;
done

Answer 6

for me it works in swift

    override func invokeTest() {
    for time in 0...15 {
        print("this test is being invoked: \(time) times")
        super.invokeTest()
    }
}