Read large files | 易学教程

问题

I tried to read a 2+ gb file in two ways, the first:

var file = File.ReadAllBytes(filepath);

returns an exception, file over 2gb.

Second way:

var file = ReadAllBytes(filepath);

public byte[] ReadAllBytes(string fileName)
{
    byte[] buffer = null;

    using (FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read))
    {
        buffer = new byte[fs.Length];
        fs.Read(buffer, 0, (int)fs.Length);
    }

    return buffer;
}

Exception: "Array dimensions exceeded supported range." My goal is to send the file in the body of http request (using WebClient class).

Any example of how to read large files?

Thanks

回答1:

You can try this:

public void ProcessLargeFile(string fileName)
{
    int bufferSize = 100 * 1024 * 1024; // 100MB
    byte[] buffer = new byte[bufferSize];
    int bytesRead = 0;

    using (FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read))
    {
        while ((bytesRead = fs.Read(buffer, 0, bufferSize)) > 0)
        {
            if (bytesRead < bufferSize)
            {
                // please note array contains only 'bytesRead' bytes from 'bufferSize'
            }

            // here 'buffer' you get current portion on file 
            // process this
        }
    }
}

That will allow you to process file by 100MB portions, you can change this value to required one.

回答2:

You are running into a rather old Limit, the 2 GiB Limit for User Mode Virtual Adress Space. You can lift it somewhat with the right compiler/manifest switch and/or running the same programm in x64 mode. You can read about it in detail here: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366778.aspx?f=255&MSPPError=-2147217396#memory_limits

At certain sizes, files or query resultsets can not be fully loaded into memory (due to the limit) or are better not fully loaded before starting processing (due to performace). For those cases, Enumerators are helpfull. Compare File.ReadLines with File.ReadAllLines. Thanks to using a Enumerator, ReadLines only needs to keep one line in memory at minimum - the current one. All Lines it already processed can be dropped. Ones that are still in the future may or may not be loaded already. At no point does it need the full file loaded into memory.

Unfortuantely it appears File.ReadAllBytes does not have a enumerator variant. It seems the different class BinaryReader does have such capability in teh form of ReadBytes.

来源：https://stackoverflow.com/questions/49256316/read-large-files

标签

file

filestream