How can I run my performance tests more than ten times?

Question

By default Xcodes performance tests are run ten times and my result is the average of those ten tests. The problem is the averaged result varies considerably each time I run it

Answer 1

In the latest Xcode (11.0+) you don't need swizzling to change iterations count. Use the following function:

func measure(options: XCTMeasureOptions, block: () -> Void)

This will allow you to specify XCTMeasureOptions which has iterationCount property.

Interesting note from docs:

A performance test runs its block iterationCount+1 times, ignoring the first iteration and recording metrics for the remaining iterations. The test ignores the first iteration to reduce measurement variance associated with “warming up” caches and other first-run behavior.

Answer 2

I typically expand the Standard Deviation value to include the range I'd accept.

One obvious workaround for you is to perform the operation 5 times within the measure block you are executing. (You'll want to change the expected time.)

    measure {
        for _ in [1...5] {
            // ...
        }
    }

Answer 3

a class dump of XCTestCase exposes this method:

- (void)_recordValues:(id)arg1 forPerformanceMetricID:(id)arg2 name:(id)arg3 unitsOfMeasurement:(id)arg4 baselineName:(id)arg5 baselineAverage:(id)arg6 maxPercentRegression:(id)arg7 maxPercentRelativeStandardDeviation:(id)arg8 maxRegression:(id)arg9 maxStandardDeviation:(id)arg10 file:(id)arg11 line:(unsigned long long)arg12;

when this method is swizzled the first parameter (arg1) has the 10 durations:

["0.003544568",
"0.003456569",
"0.003198263",
"0.003257955",
"0.003508724",
"0.003454298",
"0.003461192",
"0.00423787",
"0.003359195",
"0.003335757"]

i added 4 new values (1.0, 2.0, 3.0, 4.0) to the end of this list before passing it back to the original implementation, but unfortunately a different class that observes, XCTestLog, has an internal sanity check that gets tripped:

Assertion failure in +[XCTestLog _messageForTest:didMeasureValues:forPerformanceMetricID:name:unitsOfMeasurement:baselineName:baselineAverage:maxPercentRegression:maxPercentRelativeStandardDeviation:maxRegression:maxStandardDeviation:file:line:]
caught "NSInternalInconsistencyException", "Performance Metrics must provide 10 measurements." 

once the XCTestLog method is also overridden so it doesn't assert, the additional 4 values can be added without any complaints. unfortunately the view still only shows the 10 results.

it does however update the total time + standard deviation values in the mini view.

Before Swizzling

After Swizzling and adding 4 values

in order to view more than 10 results one would probably have to tweak the XCode runtime to tell the table to show more items.

Answer 4

The short answer: No, there is no current interface exposed to allow a measure block more than ten times.

The longer answer: No, but there is an interface exposed to modify certain metrics of the measure block. The default metrics are in a String array returned from defaultPerformanceMetrics. There appears to only be one metric supported right now: XCTPerformanceMetric_WallClockTime. This only specifies a performance metric that records the time in seconds between calls to a performance test's startMeasuring() and stopMeasuring() methods, and not the number of times the block is run.