Writing in a file from multiple threads

Question

I\'m writing a download manager in Objective-C which downloads file from multiple segments at the same times in order to improve the speed. Each segement of the file is download

Answer 1

The answers given thus far have some clear disadvantages:

• File i/o using system calls definitely has some disadvantages regarding locking.
• Caching parts in memory leads to serious issues in a memory constrained environment. (i.e. any computer)

A thread safe, efficient, lock free approach would be to use memory mapping, which works as follows:

• create the result file of (at least) the total length needed
• open() the file for read/write
• mmap() it to some place in memory. The file now "lives" in memory.
• write received parts in memory at the right offset in the file
• keep track if all pieces have been received (e.g. by posting some selector on the main thread for every piece received and stored)
• munmap() the memory and close() the file

The actual writing is handled by the kernel - your program will never issue a write system call of any form. Memory mapping generally has little downsides and is used extensively for things like shared libraries.

update: a piece of code says more than 1000 words... This is the mmap version of Mecki's lock-based multi-thread file writer. Note that writing is reduced to a simple memcpy, which cannot fail(!!), so there is no BOOL success to check. Performance is equivalent to the lock based version. (tested by writing 100 1mb blocks in parallel)

Regarding a comment on "overkill" of an mmap based approach: this uses less lines of code, doesn't require locking, is less likely to block on writing, requires no checking of return values on writing. The only "overkill" would be that it requires the developer to understand another concept than good old read/write file I/O.

The possibility to read directly into the mmapped memory region is left out, but is quite simple to implement. You can just read(fd,i_filedata+offset,length); or recv(socket,i_filedata+offset,length,flags); directly into the file.

@interface MultiThreadFileWriterMMap : NSObject
{
@private
    FILE * i_outputFile;
    NSUInteger i_length;
    unsigned char *i_filedata;
}

- (id)initWithOutputPath:(NSString *)aFilePath length:(NSUInteger)length;
- (void)writeBytes:(const void *)bytes ofLength:(size_t)length
      toFileOffset:(off_t)offset;
- (void)writeData:(NSData *)data toFileOffset:(off_t)offset;
- (void)close;
@end

#import "MultiThreadFileWriterMMap.h"
#import 
#import 

@implementation MultiThreadFileWriterMMap

- (id)initWithOutputPath:(NSString *)aFilePath length:(NSUInteger)length
{
    self = [super init];
    if (self) {
        i_outputFile = fopen([aFilePath UTF8String], "w+");
        i_length = length;
        if ( i_outputFile ) {
            ftruncate(fileno(i_outputFile), i_length);
            i_filedata = mmap(NULL,i_length,PROT_WRITE,MAP_SHARED,fileno(i_outputFile),0);
            if ( i_filedata == MAP_FAILED ) perror("mmap");
        }
        if ( !i_outputFile || i_filedata==MAP_FAILED ) {
            [self release];
            self = nil;
        }
    }
    return self;
}

- (void)dealloc
{
    [self close];
    [super dealloc];
}

- (void)writeBytes:(const void *)bytes ofLength:(size_t)length
      toFileOffset:(off_t)offset
{
    memcpy(i_filedata+offset,bytes,length);
}

- (void)writeData:(NSData *)data toFileOffset:(off_t)offset
{
    memcpy(i_filedata+offset,[data bytes],[data length]);
}

- (void)close
{
    munmap(i_filedata,i_length);
    i_filedata = NULL;
    fclose(i_outputFile);
    i_outputFile = NULL;
}

@end