Constantly growing memory allocation while fetching images over HTTP in iOS

Question

I am implementing an iOS App that needs to fetch a huge amount of images over HTTP. I've tried several approaches but independently what I do, Instuments shows constantly increasing memory allocations and the App crashes sooner or later when I run it on a device. There are no leaks shown by Instruments.

So far I have tried the following approches:

• Fetch the images using a synchronous NSURLConnection within an NSOperation
• Fetch the images using a asynchronous NSURLConnection within an NSOperation
• Fetch the images using [NSData dataWithContentsOfURL:url] in the Main-Thread
• Fetch the images using synchronous ASIHTTPRequest within an NSOperation
• Fetch the images using asynchronous ASIHTTPRequest and adding it to a NSOperationQueue
• Fetch the images using asynchronous ASIHTTPRequest and using a completionBlock

The Call Tree in Instrumetns shows that the memory is consumed while processing the HTTP-Response. In case of asynchronous NSURLConnection this is in

- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data {
[receivedData appendData:data];
}  

In case of the synchronous NSURLConnection, Instruments shows a growing CFData (store) entry. The problem with ASIHTTPRequest seems to be the same as with the asynchronous NSURLConnection in a analogous code-position. The [NSData dataWithContentsOfURL:url] approach shows an increasing amount of total memory allocation in exactely that statement.

I am using an NSAutoReleasePool when the request is done in a separate thread and I have tried to free up memory with [[NSURLCache sharedURLCache] removeAllCachedResponses] - no success.

Any ideas/hints to solve the problem? Thanks.

Edit: The behaviour only shows up if I persist the images using CoreData. Here is the code I run as a NSInvocationOperation:

-(void) _fetchAndSave:(NSString*) imageId {
NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
NSString *url = [NSString stringWithFormat:@"%@%@", kImageUrl, imageId];
HTTPResponse *response = [SimpleHTTPClient GET:url headerOrNil:nil];
NSData *data = [response payload];

if(data && [data length] > 0) {
    UIImage *thumbnailImage = [UIImage imageWithData:data];
    NSData *thumbnailData = UIImageJPEGRepresentation([thumbnailImage scaleToSize:CGSizeMake(55, 53)], 0.5); // UIImagePNGRepresentation(thumbnail); 

    [self performSelectorOnMainThread:@selector(_save:) withObject:[NSArray arrayWithObjects:imageId, data, thumbnailData, nil] waitUntilDone:NO];
}
[pool release];
}

All CoreData related stuff is done in the Main-Thread here, so there should not be any CoreData multithreading issue. However, if I persist the images, Instruments shows constantely increasing memory allocations at the positions described above.

Edit II:

CoreData related code:

-(void) _save:(NSArray*)args {
NSString *imageId = [args objectAtIndex:0];
NSData *data = [args objectAtIndex:1];
NSData *thumbnailData = [args objectAtIndex:2];

Image *image = (Image*)[[CoreDataHelper sharedSingleton] createObject:@Image];
image.timestamp =  [NSNumber numberWithDouble:[[NSDate date] timeIntervalSince1970]];
image.data = data;

Thumbnail *thumbnail = (Thumbnail*)[[CoreDataHelper sharedSingleton] createObject:@"Thumbnail"];
thumbnail.data = thumbnailData;
thumbnail.timestamp = image.timestamp;
[[CoreDataHelper sharedSingleton] save];
}

From CoreDataHelper (self.managedObjectContext is picking the NSManagedObjectContext usable in the current thread):

-(NSManagedObject *) createObject:(NSString *) entityName {
return [NSEntityDescription insertNewObjectForEntityForName:entityName inManagedObjectContext:self.managedObjectContext];
}

Answer 1

We had a similar problem. While fetching lots of images over http, there was huge growth and a sawtooth pattern in the memory allocation. We'd see the system clean up, more or less, as it went, but slowly, and not predictably. Meanwhile the downloads were streaming in, piling up on whatever was holding onto the memory. Memory allocation would crest around 200M and then we'd die.

The problem was an NSURLCache issue. You stated that you tried [[NSURLCache sharedURLCache] removeAllCachedResponses]. We tried that, too, but then tried something a little different.

Our downloads are done in groups of N images/movies, where N was typically 50 to 500. It was important that we get all of N as an atomic operation.

Before we started our group of http downloads, we did this:

NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:0 diskCapacity:0 diskPath:0];
[NSURLCache setSharedURLCache:sharedCache];

We then get each image in N over http with a synchronous call. We do this group download in an NSOperation, so we're not blocking the UI.

NSData *movieReferenceData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&error]; 

Finally, after each individual image download, and after we're done with our NSData object for that image, we call:

[sharedCache removeAllCachedResponses]; 

Our memory allocation peak behavior dropped to a very comfortable handful of megabytes, and stopped growing.

Answer 2

In this case, you're seeing exactly what you're supposed to see. -[NSMutableData appendData:] increases the size of its internal buffer to hold the new data. Since an NSMutableData is always located in memory, this causes a corresponding increase in memory usage. What were you expecting?

If the ultimate destination for these images is on disk, try using an NSOutputStream instead of NSMutableData. If you then want to display the image, you can create a UIImage pointing to the file when you're done.