Why can't my program save a large amount (>2GB) to a file?

问题

I am having trouble trying to figure out why my program cannot save more than 2GB of data to a file. I cannot tell if this is a programming or environment (OS) problem. Here is my source code:

#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#define _FILE_OFFSET_BITS 64
#include <math.h>
#include <time.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/*-------------------------------------*/
//for file mapping in Linux
#include<fcntl.h>
#include<unistd.h>
#include<sys/stat.h>
#include<sys/time.h>
#include<sys/mman.h>
#include<sys/types.h>
/*-------------------------------------*/
#define PERMS 0600
#define NEW(type) (type *) malloc(sizeof(type))
#define FILE_MODE (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH)

void write_result(char *filename, char *data, long long length){

int fd, fq;

  fd = open(filename, O_RDWR|O_CREAT|O_LARGEFILE, 0644);

  if (fd < 0) {
  perror(filename);
  return -1;
  }

  if (ftruncate(fd, length) < 0)
  {
  printf("[%d]-ftruncate64 error: %s/n", errno, strerror(errno));
  close(fd);
  return 0;
  }

  fq = write (fd, data,length);

  close(fd);

  return;

}

main()
{
long long offset = 3000000000; // 3GB
char * ttt;
ttt = (char *)malloc(sizeof(char) *offset);
printf("length->%lld\n",strlen(ttt));  // length=0
memset (ttt,1,offset);
printf("length->%lld\n",strlen(ttt));  // length=3GB
write_result("test.big",ttt,offset);
return 1;
}

According to my test, the program can generate a file large than 2GB and can allocate such large memory as well. The weird thing happened when I tried to write data into the file. I checked the file and it is empty, which is supposed to be filled with 1.

Can any one be kind and help me with this?

回答1:

You need to read a little more about C strings and what malloc and calloc do.

In your original main ttt pointed to whatever garbage was in memory when malloc was called. This means a nul terminator (the end marker of a C String, which is binary 0) could be anywhere in the garbage returned by malloc.

Also, since malloc does not touch every byte of the allocated memory (and you're asking for a lot) you could get sparse memory which means the memory is not actually physically available until it is read or written.

calloc allocates and fills the allocated memory with 0. It is a little more prone to fail because of this (it touches every byte allocated, so if the OS left the allocation sparse it will not be sparse after calloc fills it.)

Here's your code with fixes for the above issues. You should also always check the return value from write and react accordingly. I'll leave that to you...

main()
{
    long long offset = 3000000000; // 3GB
    char * ttt;
    //ttt = (char *)malloc(sizeof(char) *offset);
    ttt = (char *)calloc( sizeof( char ), offset ); // instead of malloc( ... )
    if( !ttt )
    {
        puts( "calloc failed, bye bye now!" );
        exit( 87 );
    }

    printf("length->%lld\n",strlen(ttt));  // length=0  (This now works as expected if calloc does not fail)
    memset( ttt, 1, offset );
    ttt[offset - 1] = 0;  // Now it's nul terminated and the printf below will work
    printf("length->%lld\n",strlen(ttt));  // length=3GB
    write_result("test.big",ttt,offset);
    return 1;
}

Note to Linux gurus... I know sparse may not be the correct term. Please correct me if I'm wrong as it's been a while since I've been buried in Linux minutiae. :)

回答2:

Looks like you're hitting the internal file system's limitation for the iDevice: ios - Enterprise app with more than resource files of size 2GB

2Gb+ files are simply not possible. If you need to store such amount of data you should consider using some other tools or write the file chunk manager.

回答3:

I'm going to go out on a limb here and say that your problem may lay in memset().

The best thing to do here is, I think, after memset() ing it,

for (unsigned long i = 0; i < 3000000000; i++) {
  if (ttt[i] != 1) { printf("error in data at location %d", i); break; }
}

Once you've validated that the data you're trying to write is correct, then you should look into writing a smaller file such as 1GB and see if you have the same problems. Eliminate each and every possible variable and you will find the answer.

来源：https://stackoverflow.com/questions/11169202/why-cant-my-program-save-a-large-amount-2gb-to-a-file

标签

large-files