Fastest way to create large file in c++?

Question

Create a flat text file in c++ around 50 - 100 MB with the content 'Added first line' should be inserted in to the file for 4 million times

Answer 1

using old style file io

fopen the file for write.

fseek to the desired file size - 1.

fwrite a single byte

fclose the file

Answer 2

The fastest way to create a file of a certain size is to simply create a zero-length file using creat() or open() and then change the size using chsize(). This will simply allocate blocks on the disk for the file, the contents will be whatever happened to be in those blocks. It's very fast since no buffer writing needs to take place.

Answer 3

Not sure I understand the question. Do you want to ensure that every character in the file is a printable ASCII character? If so, what about this? Fills the file with "abcdefghabc...."

#include <stdio.h>
int main ()
{
   const int FILE_SiZE = 50000; //size in KB
   const int BUFFER_SIZE = 1024;
   char buffer [BUFFER_SIZE + 1];
   int i;
   for(i = 0; i < BUFFER_SIZE; i++)
      buffer[i] = (char)(i%8 + 'a');
   buffer[BUFFER_SIZE] = '\0';

   FILE *pFile = fopen ("somefile.txt", "w");
   for (i = 0; i < FILE_SIZE; i++)
     fprintf(pFile, buffer);

   fclose(pFile);

   return 0;
}

Answer 4

You haven't mentioned the OS but I'll assume creat/open/close/write are available.

For truly efficient writing and assuming, say, a 4k page and disk block size and a repeated string:

1. open the file.
2. allocate 4k * number of chars in your repeated string, ideally aligned to a page boundary.
3. print repeated string into the memory 4k times, filling the blocks precisely.
4. Use write() to write out the blocks to disk as many times as necessary. You may wish to write a partial piece for the last block to get the size to come out right.
5. close the file.

This bypasses the buffering of fopen() and friends, which is good and bad: their buffering means that they're nice and fast, but they are still not going to be as efficient as this, which has no overhead of working with the buffer.

This can easily be written in C++ or C, but does assume that you're going to use POSIX calls rather than iostream or stdio for efficiency's sake, so it's outside the core library specification.

Answer 5

I faced the same problem, creating a ~500MB file on Windows very fast. The larger buffer you pass to fwrite() the fastest you'll be.

int i;
FILE *fp;

fp = fopen(fname,"wb");

if (fp != NULL) {

    // create big block's data
    uint8_t b[278528]; // some big chunk size

    for( i = 0; i < sizeof(b); i++ ) // custom initialization if != 0x00
    {
        b[i] = 0xFF;
    }

    // write all blocks to file
    for( i = 0; i < TOT_BLOCKS; i++ )
        fwrite(&b, sizeof(b), 1, fp);

    fclose (fp);
}

Now at least on my Win7, MinGW, creates file almost instantly. Compared to fwrite() 1 byte at time, that will complete in 10 Secs. Passing 4k buffer will complete in 2 Secs.

Answer 6

Fastest way to create large file in c++? Ok. I assume fastest way means the one that takes the smallest run time.

Create a flat text file in c++ around 50 - 100 MB with the content 'Added first line' should be inserted in to the file for 4 million times.

preallocate the file using old style file io

fopen the file for write.
fseek to the desired file size - 1.
fwrite a single byte
fclose the file

create a string containing the "Added first line\n" a thousand times.
find it's length.

preallocate the file using old style file io

fopen the file for write.
fseek to the the string length * 4000
fwrite a single byte
fclose the file

open the file for read/write
loop 4000 times, 
    writing the string to the file.
close the file.

That's my best guess. I'm sure there are a lot of ways to do it.