Generate gcda-files with Xcode5, iOS7 simulator and XCTest

Question

Being inspired by the solution to this question I tried using the same approach with XCTest.

I\'ve set \'Generate Test Coverage Files=YES\' and \'Instrument Program

Answer 1

GCOV Flush in -(void)applicationWillTerminate didn't work for me, I think because my App is running in Background.

I also set 'Generate Test Coverage Files=YES' and 'Instrument Program Flow=YES' but no gcda-Files.

Then I executed "__gcov_flush()" in -(void)tearDown from the TestClass, which gave me gcda-Files for my TestClass ;)

Then I created the following Function in my AppDelegate:

@interface AppDelegate : UIResponder <UIApplicationDelegate>
+(void)gcovFlush;
@end

@implementation AppDelegate
+(void)gcovFlush{
  extern void __gcov_flush(void);
  __gcov_flush();
  NSLog(@"%s - GCOV FLUSH!", __PRETTY_FUNCTION__);
}
@end

I called [AppDelegate gcovFlush] in my -(void)tearDown and voilá, there are my gcda Files ;)

I hope this helps, bye Chris

Answer 2

- (void)applicationWillTerminate:(UIApplication*)application must be defined in your application delegate, not in the observer class.

I didn't have any library problems. "-lgov" is not needed and you don't have to add any libraries. Coverage is supported directly by LLVM compiler.

Answer 3

Because you have to create a new XCTestSuiteRun instance in the testSuiteDidStop method, you are not going to get the proper results on an == check. Instead of depending on instance equality, we used a simple counter and call flush when it hits zero, which it will when the top-level XCTestSuite finishes executing. There are probably more clever ways to do this.

First, we had to set 'Generate Test Coverage Files=YES' and 'Instrument Program Flow=YES' in both the Test and main app targets.

#import <XCTest/XCTestLog.h>
#import <XCTest/XCTestSuiteRun.h>
#import <XCTest/XCTest.h>

// Workaround for XCode 5 bug where __gcov_flush is not called properly when Test Coverage flags are set

@interface GCovrTestObserver : XCTestLog
@end

#ifdef DEBUG
extern void __gcov_flush(void);
#endif

static NSUInteger sTestCounter = 0;
static id mainSuite = nil;

@implementation GCovrTestObserver

- (void)testSuiteDidStart:(XCTestRun *)testRun {
    [super testSuiteDidStart:testRun];

    XCTestSuiteRun *suite = [[XCTestSuiteRun alloc] init];
    [suite addTestRun:testRun];

    sTestCounter++;

    if (mainSuite == nil) {
        mainSuite = suite;
    }
}

- (void)testSuiteDidStop:(XCTestRun *)testRun {

    sTestCounter--;

    [super testSuiteDidStop:testRun];

    XCTestSuiteRun *suite = [[XCTestSuiteRun alloc] init];
    [suite addTestRun:testRun];

    if (sTestCounter == 0) {
        __gcov_flush();
    }
}

@end

There was an additional step required, because the +initialize call was not being made on the observer when included in the Test target.

In the AppDelegate, add the following:

#ifdef DEBUG
+(void) initialize {
    if([self class] == [AppDelegate class]) {
        [[NSUserDefaults standardUserDefaults] setValue:@"GCovrTestObserver"
                                                 forKey:@"XCTestObserverClass"];
    }
}
#endif

Answer 4

Update 1:

After reading a bit more about this, 2 things have now become clear to me (emphasis added):

Tests and the tested application are compiled separately. Tests are actually injected into the running application, so the __gcov_flush() must be called inside the application not inside the tests.

— Xcode5 Code Coverage (from cmd-line for CI builds) - Stack Overflow

and,

Again: Injection is complex. Your take away should be: Don’t add .m files from your app to your test target. You’ll get unexpected behavior.

— Testing View Controllers – #1 – Lighter View Controllers

The code below was changed to reflect these two insights…

---

Update 2:

Added information on how to make this work for static libraries, as requested by @MdaG in the comments. The main changes for libraries is that:

• We can flush directly from the -stopObserving method because there isn't a separate app where to inject the tests.

• We must register the observer in the +load method because by the time the +initialize is called (when the class is first accessed from the test suite) it's already too late for XCTest to pick it up.

---

Solution

The other answers here have helped me immensely in setting up code coverage in my project. While exploring them, I believe I've managed to simplify the code for the fix quite a bit.

Considering either one of:

• ExampleApp.xcodeproj created from scratch as an "Empty Application"
• ExampleLibrary.xcodeproj created as an independent "Cocoa Touch Static Library"

These were the steps I took to enable Code Coverage generation in Xcode 5:

1. Create the GcovTestObserver.m file with the following code, inside the ExampleAppTests group:

   #import <XCTest/XCTestObserver.h>
   
   @interface GcovTestObserver : XCTestObserver
   @end
   
   @implementation GcovTestObserver
   
   - (void)stopObserving
   {
       [super stopObserving];
       UIApplication* application = [UIApplication sharedApplication];
       [application.delegate applicationWillTerminate:application];
   }
   
   @end
   

   When doing a library, since there is no app to call, the flush can be invoked directly from the observer. In that case, add the file to the ExampleLibraryTests group with this code instead:

   #import <XCTest/XCTestObserver.h>
   
   @interface GcovTestObserver : XCTestObserver
   @end
   
   @implementation GcovTestObserver
   
   - (void)stopObserving
   {
       [super stopObserving];
       extern void __gcov_flush(void);
       __gcov_flush();
   }
   
   @end
   

2. To register the test observer class, add the following code to the @implementation section of either one of:

   • ExampleAppDelegate.m file, inside the ExampleApp group
   • ExampleLibrary.m file, inside the ExampleLibrary group

    

   #ifdef DEBUG
   + (void)load {
       [[NSUserDefaults standardUserDefaults] setValue:@"XCTestLog,GcovTestObserver"
                                                forKey:@"XCTestObserverClass"];
   }
   #endif
   

   Previously, this answer suggested to use the +initialize method (and you can still do that in case of Apps) but it doesn't work for libraries…

   In the case of a library, the +initialize will probably be executed only when the tests invoke the library code for the first time, and by then it's already too late to register the observer. Using the +load method, the observer registration in always done in time, regardless of which scenario.

3. In the case of Apps, add the following code to the @implementation section of the ExampleAppDelegate.m file, inside the ExampleApp group, to flush the coverage files on exiting the app:

   - (void)applicationWillTerminate:(UIApplication *)application
   {
   #ifdef DEBUG
       extern void __gcov_flush(void);
       __gcov_flush();
   #endif
   }
   

4. Enable Generate Test Coverage Files and Instrument Program Flow by setting them to YES in the project build settings (for both the "Example" and "Example Tests" targets).

   To do this in an easy and consistent way, I've added a Debug.xcconfig file associated with the project's "Debug" configuration, with the following declarations:

   GCC_GENERATE_TEST_COVERAGE_FILES = YES
   GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = YES
   

5. Make sure all the project's .m files are also included in the "Compile Sources" build phase of the "Example Tests" target. Don't do this: app code belongs to the app target, test code belongs to the test target!

After running the tests for your project, you'l be able to find the generated coverage files for the Example.xcodeproj in here:

cd ~/Library/Developer/Xcode/DerivedData/
find ./Example-* -name *.gcda

Notes

Step 1

The method declaration inside XCTestObserver.h indicates:

/*! Sent immediately after running tests to inform the observer that it's time 
    to stop observing test progress. Subclasses can override this method, but 
    they must invoke super's implementation. */
- (void) stopObserving;

Step 2

2.a)

By creating and registering a separate XCTestObserver subclass, we avoid having to interfere directly with the default XCTestLog class.

The constant key declaration inside XCTestObserver.h suggests just that:

/*! Setting the XCTestObserverClass user default to the name of a subclass of 
    XCTestObserver indicates that XCTest should use that subclass for reporting 
    test results rather than the default, XCTestLog. You can specify multiple 
    subclasses of XCTestObserver by specifying a comma between each one, for 
    example @"XCTestLog,FooObserver". */
XCT_EXPORT NSString * const XCTestObserverClassKey;

2.b)

Even though it's common practice to use if(self == [ExampleAppDelegate class]) around the code inside +initialize [Note: it's now using +load], I find it easier to omit it in this particular case: no need to adjust to the correct class name when doing copy & paste.

Also, the protection against running the code twice isn't really necessary here: this is not included in the release builds, and even if we subclass ExampleAppDelegate there is no problem in running this code more than one.

2.c)

In the case of libraries, the first hint of the problem came from this code comment in the Google Toolbox for Mac project: GTMCodeCovereageApp.m

+ (void)load {
  // Using defines and strings so that we don't have to link in XCTest here.
  // Must set defaults here. If we set them in XCTest we are too late
  // for the observer registration.
  // (...)

And as the NSObject Class Reference indicates:

initialize — Initializes the class before it receives its first message

load — Invoked whenever a class or category is added to the Objective-C runtime

The “EmptyLibrary” project

In case someone tries to replicate this process by creating their own "EmptyLibrary" project, please bear in mind that you need to invoke the library code from the default emtpy tests somehow.

If the main library class is not invoked from the tests, the compiler will try to be smart and it won't add it to the runtime (since it's not being called anywhere), so the +load method doesn't get called.

You can simply invoke some harmless method (as Apple suggests in their Coding Guidelines for Cocoa # Class Initialization). For example:

- (void)testExample
{
    [ExampleLibrary self];
}

Answer 5

Here's another solution that avoids having to edit your AppDelegate

UIApplication+Instrumented.m (put this in your main target):

@implementation UIApplication (Instrumented)

#ifdef DEBUG

+ (void)load
{
    NSString* key = @"XCTestObserverClass";
    NSString* observers = [[NSUserDefaults standardUserDefaults] stringForKey:key];
    observers = [NSString stringWithFormat:@"%@,%@", observers, @"XCTCoverageFlusher"];
    [[NSUserDefaults standardUserDefaults] setValue:observers forKey:key];
}

- (void)xtc_gcov_flush
{
    extern void __gcov_flush(void);
    __gcov_flush();
}

#endif

@end

XCTCoverageFlusher.m (put this in your test target):

@interface XCTCoverageFlusher : XCTestObserver
@end

@implementation XCTCoverageFlusher

- (void) stopObserving
{
    [super stopObserving];
    UIApplication* application = [UIApplication sharedApplication];
    SEL coverageFlusher = @selector(xtc_gcov_flush);
    if ([application respondsToSelector:coverageFlusher])
    {
        objc_msgSend(application, coverageFlusher);
    }
    [application.delegate applicationWillTerminate:application];
}

@end

Answer 6

The process for this is a little different if you're using Specta, since it does its own swizzling. The following is working for me:

Test Bundle:

@interface MyReporter : SPTNestedReporter // keeps the default reporter style
@end

@implementation MyReporter

- (void) stopObserving
{
  [super stopObserving];
  UIApplication* application = [UIApplication sharedApplication];
  [application.delegate applicationWillTerminate:application];
}

@end

AppDelegate:

- (void)applicationWillTerminate:(UIApplication *)application
{
#ifdef DEBUG
  extern void __gcov_flush(void);
  __gcov_flush();
#endif
}

You then need to enable your custom reporter subclass by setting the environmental variable SPECTA_REPORTER_CLASS to MyReporter in the Run section of your main scheme.