Core Data Design - How to have both app data and user data?

Question

I have a situation where I have a base set of information that ships with my app. The user can add or edit this information. But from time to time, I need to be able to upda

Answer 1

So from what I can tell you have 4 requirements (which, really, could be 4 different questions)

1. Application data and user data should be kept separate.
2. Application ships with a "base" set of data
3. Application "base" data will be editable by user.
4. Application "base" data will need to be periodically updated, but can't conflict with user changes.

Application data and user data should be kept separate.

The first requirement, separation of application and user data, relates to the iOS Data Storage Guidelines. Not following the data storage guidelines correctly can lead to an app store rejection (2.23: Apps must follow the iOS Data Storage Guidelines or they will be rejected) with a cryptic response pointing you to Technical Q&A 1719. The data storage guidelines exist for several reasons - controlling the size of backups, policies for automatic purging of files when device space is low, and general guidelines for what goes where (developers were getting this wrong).

Data created or edited by the user belongs under <Application Sandbox>/Documents. This corresponds to NSDocumentDirectory.

Data that can be regenerated on demand goes under <Application Sanbox>/Library/Caches. This is typically downloaded data or cache files. iOS is in charge of automatically purging anything under this directory. This corresponds to NSCachesDirectory. Data can also be stored in <Application Sandbox>/tmp, though in this case it's recommended that your application periodically purge these files. This corresponds to NSTemporaryDirectory().

Application data - data that the application needs, and may (or may not) be able to easily recreate - belongs under <Application Sandbox>/Library/Application Support and should also be marked with the "Do not back up" (NSURLIsExcludedFromBackupKey) attribute. When a file is marked with that attribute it will not be backed up AND the system will not purge the data automatically. Because of this, it's recommended that your application at least attempt to control the growth of these files and purge them when not needed. This directory corresponds to NSApplicationSupportDirectory, and the convention is to create a subdirectory with your bundle identifier, and additional subdirectories under that.

What does this mean for a Core Data application?

User data and application data should use different store files, in different locations. You will still use a single NSPersistentStoreCoordinator, but add two different persistent stores to it. Your managed object model will need two different configurations - each store gets it's own configuration, and each entity is attached to one of the configurations.

Configuration:

This will drive the design of your data model - you should not have a single entity type exist in two different stores, which means you will not be able to have "user edited Foo entity" exist in the user store, while "application provided Foo entity" exists in the application store - unless Foo is abstract, and each store has it's own concrete entity (this is just one possible solution). Cross store relationships can be implemented as fetched properties (more on that in a bit).

Since Core Data SQLite persistent stores are not single files but collections of files, it's recommended that each store has it's own directory for the store files - this makes saving, deleting, backing up, app updates, etc. much more efficient and reliable. With that in mind, your file structure should look like:

Application store URL: <Application Sandbox>/Library/Application Support/com.yourcompany.YourApp/ApplicationData/Application.sqlite You will have to create the directories com.yourcompany.YourApp/ApplicationData, and will need to set NSURLIsExcludedFromBackupKey on them.

User store URL: <Application Sandbox>/Documents/UserData/User.sqlite You will need to create the directory UserData.

This sounds like a lot of work, but it isn't:

- (NSURL *)supportFilesDirectoryURL {
    NSURL       *result                         = nil;
    NSURL       *applicationSupportDirectoryURL = [[[NSFileManager defaultManager] URLsForDirectory:NSApplicationSupportDirectory inDomains:NSUserDomainMask] lastObject];
    NSString    *bundleName                     = [[NSBundle bundleForClass:[self class]] bundleIdentifier];

    result = [applicationSupportDirectoryURL URLByAppendingPathComponent:bundleName isDirectory:@YES];

    return result;
}

- (NSURL *) applicationStoreURL {
    NSError             *error                  = nil;
    NSFileCoordinator   *coordinator            = nil;
    __block BOOL        didCreateDirectory      = NO;
    NSURL               *supportDirectoryURL    = [self supportFilesDirectoryURL];
    NSURL               *storeDirectoryURL      = [supportDirectoryURL URLByAppendingPathComponent:@"ApplicationData" isDirectory:YES];

    coordinator = [[NSFileCoordinator alloc] initWithFilePresenter:nil];
    [coordinator coordinateWritingItemAtURL:storeDirectoryURL options:NSFileCoordinatorWritingForDeleting error:&error byAccessor:^(NSURL *writingURL){
        NSFileManager   *fileManager    = [[NSFileManager alloc] init];
        NSError         *fileError      = nil;
        if (![fileManager createDirectoryAtURL:writingURL withIntermediateDirectories:YES attributes:nil error:&fileError]){
            [[NSOperationQueue mainQueue] addOperationWithBlock:^{
                // Handle the error
            }];
        } else {
            // Setting NSURLIsExcludedFromBackupKey on the directory will exclude all items in this directory
            // from backups. It will also prevent them from being purged in low space conditions. Because of this,
            // the files inside this directory should be purged by the application.
            [writingURL setResourceValue:@YES forKey:NSURLIsExcludedFromBackupKey error:&fileError];
            didCreateDirectory = YES;
        }
    }];

    // See NSFileCoordinator.h for an explanation.
    if (didCreateDirectory == NO){
        [[NSOperationQueue mainQueue] addOperationWithBlock:^{
            // Handle the error.
        }];
    }

    return [NSURL URLWithString:@"Application.sqlite" relativeToURL:storeDirectoryURL ];
}

You have to add both stores to your persistent store coordinator, each with the correct configuration name as defined in your managed object model:

    if (![persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:@"Application" URL:[self applicationStoreURL] options:nil error:&error]) {
        [[NSOperationQueue mainQueue] addOperationWithBlock:^{
            // Handle the error.
        }];
    }

    if (![persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:@"User" URL:[self userStoreURL] options:nil error:&error]) {
        [[NSOperationQueue mainQueue] addOperationWithBlock:^{
            // Handle the error.
        }];
    }

Unfortunately, fetched properties have a problem with nested contexts.

If the context used is not the root context use of a fetched property will crash - you should duplicate this radar to ensure this gets fixed in a future release. Fetched properties are critical for conforming to the data storage guidelines.

Application ships with a "base" set of data

There are a number of ways to handle this:

1. Create managed objects from JSON/XML/property list/CSV/etc. data and insert into a managed object context and save.
2. Start with a store as a read only store in addition to your user and application stores.
3. Start with a set of pre-built SQLite files (such as from a command line tool) as a read only store in addition to your user and application stores.
4. Start with a pre-built store as in 3, but perform a migration to move those objects into another store.

Each has advantages and disadvantages:

1. This can be slow, complicated, difficult to maintain, and error prone. This is one of the more common ways to do this, and one of the most sucky. Seriously. But if this is what you're most comfortable maintaining, more power to you.
2. Remember when I said earlier that a given entity should not exist in more than one store/configuration? Well, I didn't tell you the whole truth. A given entity should only exist in a single writable store. For our Foo entity, we can have one writable store, but as many read only stores as we want. If the store is added to the persistent store coordintor with the option NSReadOnlyPersistentStoreOption, it will be read-only. This means that you can use a pre-built NSSQLiteStore that is read-only and contains your "starter" data, or you can be more ambitious and use an NSAtomicStore or NSIncrementalStore implementation that can read some other data format. This is the least error prone solution, and one of the easiest to maintain.
3. To create a set of SQLite files for this you will need to build a tool that shares much of the Core Data code and managed object model as your application. Otherwise this is the same as 3.
4. You can perform a migration between a store such as one used in options 2 or 3, and use a migration to move data into your application or user store. Migrations are typically much more performant than doing something like option 1, though they have some limitations: for example, a lightweight migration can work very well, but will do nothing to prevent duplicates or conflicts. A custom migration would be required. Such a migration would look like: sqlStore = [persistentStoreCoordinator migratePersistentStore:store toURL:[self applicationStoreURL] options:nil withType:NSSQLiteStoreType error:&error]; where store is the source store already added to the coordinator (this operation will remove it, and add the migrated store). This effectively injects the contents of store into the NSSQLiteStoreType store at applicationStoreURL.

Application "base" data will be editable by user.

This will depend very much on the design of your data model and what you're trying to accomplish. I can't really make generalizations about this. Because the application and user data exist in separate stores, and because of the data storage guidelines (and their intent), it's difficult to do exactly what you are asking for. A well designed data model, however, should make this easy. What are the users changing? How can you make that "belong" to them? A good example here is "favoriting". The content the user is favoriting - the product inventory - should not change, and the "favorite" belongs to the user. They're not editing the book/music/widget. They're creating a relationship between themselves and whatever identifies that content. Those kinds of use cases are where fetched properties really shine.

Application "base" data will need to be periodically updated, but can't conflict with user changes.

Annnnd this is where it all falls apart. This is pretty difficult because of the previous requirements, and that we don't know much about the data model or what we're trying to accomplish with it.

In general, you don't want to be updating/importing anything that can conflict with the data being changed on the device. The upside here is that with clean separation between the user and application data as described above, changing application data should be easy because the user won't be changing it. You could throw out all of the application data and replace it, and if you data model is designed well it will just work.

But let's say you're saying data from a remote source and importing it over changes made on the device. Every once in a while you download a bunch of JSON with product updates. The JSON is parsed, managed objects are created, and those are saved. You're almost certainly going to get conflicts at some point. SOME change made on the device at the object or property level is going to conflict with the new information being imported. You can use an NSMergePolicy to handle the conflicts, but in this specific scenario a merge policy may not be adequate. Which set of information is "correct"? The information on the device, or the remote information? Do you accept the new data blindly, or do you have to check it property by property? People do this. I'm not kidding. You'd think they would have learned by now.

Good separation of user and application data helps solve this (and other) problems you will run into.