Core Data memory usage while importing large dataset

Question

I\'m now stuck for about two weeks with a nasty Core Data problem. I read lots of blogpost, articles and SO questions/answers but I\'m still not able to solve my problem.

<

Answer 1

I have an import helper that does something very similar.

Have a look at the code below and see if it helps you

__block NSUInteger i = 0;
NSArray *jsonArray = ...
for (NSDictionary *dataStucture in jsonArray)
{
    [managedObjectContext performBlock:^{
        @autoreleasepool {
            i++;
            A *a = (A*)[self newManagedObjectOfType:@"A" inManagedObjectContext:managedObjectContext];
            [self parseData:[dataStucture objectForKey:@"a"]
                 intoObject:a
     inManagedObjectContext:managedObjectContext];

            [managedObjectContext refreshObject:a
                                   mergeChanges:YES];
            if (i > 20) // Arbitrary number here
            {
                NSError *error = nil;
                [managedObjectContext save:&error];
                [managedObjectContext reset];
            }

            [managedObjectContext refreshObject:a
                                   mergeChanges:YES];

        }
        dispatch_async(dispatch_get_main_queue(), ^{
            [self saveMainThreadManagedObjectContext];

            NSLog(@"DONE");
            // parsing is done, now you see that there are still
            // A's, B's and C's left in memory.
            // Every managedObjectContext is saved and no references are kept
            // to any A, B and C so they should be released. This is not true,
            // so a managedObjectContext is keeping a strong reference to these
            // objects.
        });
    }];
}

Answer 2

For every NSManagedObjectContext that you keep around for a specific purpose you are going to accumulate instances of NSManagedObject

A NSManagedObjectContext is just a piece of scratch note paper that you can instantiate at will and save if you wish to keep changes in the NSPersistentStore and then discard afterward.

For the parsing operations (layer 3) try creating a MOC for the op , do your parsing, save the MOC and then discard it afterwards.

It feels like you have at least one layer of MOC being held in strong references too many.

Basically ask the question for each of the MOC's. "Why am keeping this object and its associated children alive".

Answer 3

Robin,

I have a similar problem which I solved differently than you have. In your case, you have a third, IMO, redundant MOC, the parent MOC. In my case, I let the two MOCs communicate, in an old school fashion, through the persistent store coordinator via the DidSave notifications. The new block oriented APIs make this much simpler and robust. This lets me reset the child MOCs. While you gain a performance advantage from your third MOC, it isn't that great of an advantage over the SQLite row cache which I exploit. Your path consumes more memory. Finally, I can, by tracking the DidSave notifications, trim items as they are created.

BTW, you are also probably suffering from a massive increase in the size of your MALLOC_TINY and MALLOC_SMALL VM regions. My trailing trimming algorithm lets the allocators reuse space sooner and, hence, retards the growth of these problematic regions. These regions are, in my experience, due to their large resident memory footprint a major cause for my app, Retweever, being killed. I suspect your app suffers the same fate.

When the memory warnings come, I call the below snippet:

[self.backgroundMOC performBlock: ^{ [self.backgroundMOC reset]; }];

[self.moc save];

[self.moc.registeredObjects trimObjects];

-[NSArray(DDGArray) trimObjects] just goes through an array and refreshes the object, thus trimming them.

In summary, Core Data appears to implement a copy on write algorithm for items that appear in many MOCs. Hence, you have things retained in unexpected ways. I focus upon breaking these connections after import to minimize my memory footprint. My system, due to the SQLite row cache, appears to performa acceptably well.

Andrew