ifstream:: What is the maximum file size that a ifstream can read

问题

I tried to read a 3GB data file using ifstream and it gives me wrong file size, whereas when I read a 600MB file, it gives me the correct result. In addition to wrong file size, I am also unable to read the entire file using ifstream.

Here is the code that I used

        std::wstring name;
        name.assign(fileName.begin(), fileName.end());
        __stat64 buf;
        if (_wstat64(name.c_str(), &buf) != 0)
            std::cout << -1; // error, could use errno to find out more

        std::cout << " Windows file size : " << buf.st_size << std::endl;;


        std::ifstream fs(fileName.c_str(), std::ifstream::in | std::ifstream::binary);
        fs.seekg(0, std::ios_base::end);

        std::cout << " ifstream  file size: " << fs.tellg() << std::endl;

The output for 3GB file was

 Windows file size : 3147046042
 ifstream  file size: -1147921254

Whereas the output for 600 MB file was

 Windows file size : 678761111
 ifstream  file size: 678761111

Just in case, I also tested for 5GB file and 300 MB file,

The output for 5GB file was

Windows file size : 5430386900
 ifstream  file size: 1135419604

The output for 300MB file was

Windows file size : 318763632
 ifstream  file size: 318763632

It looks to me like it is reaching some limit.

I am testing the code using Visual Studio 2010 on a Windows Machine which has plenty of memory and disk space.

I am trying to read some large files. Which is a good stream reader to use if ifstream can't read large files?

回答1:

I think you want to say:

std::cout << " ifstream  file size: " << fs.tellg().seekpos() << std::endl;

At least that works correctly for a 6GB file I have laying around. But I'm compiling with Visual Studio 2012. And even your original code works fine too on that environment.

So I suspect that this is an bug in the std library on VS 2010 that got fixed in VS 2012. Whether it's a bug in the operator overloading for the pos_type or if that class isn't 64-bit aware is unknown. I'd have to install VS 2010 to validate, but that is likely the problem.

回答2:

I modified your code slightly, to something that would compile:

#include <fstream>
#include <iostream>
#include <string>
#include <windows.h>

int main() { 

    std::wstring name(L"whatever.txt");

    __stat64 buf;
    if (_wstat64(name.c_str(), &buf) != 0)
        std::cout << -1; // error, could use errno to find out more

    std::cout << " Windows file size : " << buf.st_size << std::endl;;


    std::ifstream fs(name.c_str(), std::ifstream::in | std::ifstream::binary);
    fs.seekg(0, std::ios_base::end);

    std::cout << " ifstream  file size: " << fs.tellg() << std::endl;

    return 0;
}

I tried this on a ~3 Gigabyte file. With VS 2012 (either 32- or 64-bit) it produced:

 Windows file size : 3581853696
 ifstream  file size: 3581853696

With 32-bit VS 2008 (sorry, don't have a copy of VS 2010 installed right now) I got:

 Windows file size : 3581853696
 ifstream  file size: -713113600

So, it would appear that older versions of VS/VC++ used a 32-bit signed number for file sizes, so their practical limit for iostreams was probably 2 gigabytes. With VS 2012, that has apparently been corrected.

回答3:

The maximum file sizes are determined by the compiler and the OS.

The compiler has control over the sizes of the variables used to access the file sizes.

The OS determines the largest file size that it can support.

The C++ language does not limit the file size.

Example 1:
The compiler could allocate 16 bits for the file position variable, while the OS may use a 32-bit pointer for the maximum file size. In this case, the compiler is the limiting factor.

Example 2:
The compiler could use 32-bits for the file position variable, but the OS uses 24 bits. In this example, the OS is the limiting factor.

In summary, the maximum file size depends on both the compiler and the OS.

来源：https://stackoverflow.com/questions/16324811/ifstream-what-is-the-maximum-file-size-that-a-ifstream-can-read

标签

c++

stl

ifstream