Byte[stream.length] - out of memory exception, best way to solve?

问题

i am trying to read the stream of bytes from a file. However when I try to read the bytes I get a

The function evaluation was disabled because of an out of memory exception

Quite straightforward. However, what is the best way of getting around this problem? Is it too loop around the length at 1028 at a time? Or is there a better way?

The C# I am using

BinaryReader br = new BinaryReader(stream fs);

// The length is around 600000000
long Length = fs.Length;

// Error here
bytes = new byte[Length];

for (int i = 0; i < Length; i++)
{
   bytes [i] = br.ReadByte();
}

Thanks

回答1:

Well. First of all. Imagine a file with the size of e.g. 2GB. Your code would allocate 2GB of memory. Just read the part of the file you really need instead of the whole file at once. Secondly: Don't do something like this:

for (int i = 0; i < Length; i++)
{
   bytes [i] = br.ReadByte();
}

It is quite inefficient. To read the raw bytes of a stream you should use something like this:

using(var stream = File.OpenRead(filename))
{
    int bytesToRead = 1234;
    byte[] buffer = new byte[bytesToRead];

    int read = stream.Read(buffer, 0, buffer.Length);

    //do something with the read data ... e.g.:
    for(int i = 0; i < read; i++)
    {
        //...
    }
}

回答2:

When you try to allocate an array, the CLR lays it out contiguously in virtual memory given to it by the OS. Virtual memory can be fragmented, however, so a contiguous 1 GB block may not be available, hence the OutOfMemoryException. It doesn't matter how much physical RAM your machine has, and this problem is not limited to managed code (try allocating a huge array in native C and you'll find similar results).

Instead of allocating a huge array, I recommend using several smaller arrays, an ArrayList or List, that way the Framework can allocate the data in chunks.

Hope that helps

回答3:

I believe that the instantiation of the stream object already reads the file (into a cache). Then your loop copies the bytes in memory to another array.

So, why not to use the data into "br" instead of making a further copy?

来源：https://stackoverflow.com/questions/24141948/bytestream-length-out-of-memory-exception-best-way-to-solve

标签

stream

binaryreader