Memory issues with InputStream in Java

Question

I need to read a file into an array of Bytes.The entire file needs to be read into the array. The problem is I am getting an OutOfMemory Error since the file size is too large.

Answer 1

You need to have far more free memory than the largest file you can have to use this approach. Given a machine with 24 GB costs less than £2K, this isn't as silly idea as it used to be. Actually the 2GB limit for a byte[] is more of a headache in some situations.

However, the usual way to read an InputStream is to read a block of say 8KB at a time. This way you only need to have far more than 8KB free.

BTW: You can use -mx1g instead of the option you are using.

Answer 2

The workarround would be, not to load the entire file into the RAM. In fact you can't do this for large files, because you have to allocate lots of memory at one peace, which may not work.

The question ist: Do you really need the entire file in memory?

edit

InputStream in = new FileInputStream(file);
long length = file.length();     
// At this point a warning should appear, because the code would
// not work for files larger than Integer.MAX_VALUE                   
byte[] out = new byte[(int)length]; 

Answer 3

No, if your file is too large to fit into memory, it's too large to fit in memory.

A better solution would be to try to process the stream as a stream, rather than loading the whole thing into memory. Without knowing what processing you're trying to achieve, we can't really tell whether or not that's feasible.

For example, if you're just trying to compute a secure hash of the file, you should be able to do that without loading significant amounts of data in at a time - but if your processing requires random access to data, you may need to use RandomAccessFile instead.

Answer 4

How about using a memory mapped file: FileChannel

From http://www.java-tips.org/java-se-tips/java.nio/how-to-create-a-memory-mapped-file-3.html:

try {
    File file = new File("filename");

    // Create a read-only memory-mapped file
    FileChannel roChannel = 
      new RandomAccessFile(file, "r").getChannel();

    ByteBuffer readonlybuffer = 
      roChannel.map(FileChannel.MapMode.READ_ONLY, 
0, (int)roChannel.size());

    // Create a read-write memory-mapped file
    FileChannel rwChannel = 
      new RandomAccessFile(file, "rw").getChannel();

    ByteBuffer writeonlybuffer= 
      rwChannel.map(FileChannel.MapMode.READ_WRITE, 
  0, (int)rwChannel.size());

    // Create a private (copy-on-write) memory-mapped file.
    // Any write to this channel results in a private 
    // copy of the data.
    FileChannel pvChannel = 
      new RandomAccessFile(file, "rw").getChannel();

    ByteBuffer privatebuffer = 
      roChannel.map(FileChannel.MapMode.READ_WRITE, 
  0, (int)rwChannel.size());

} catch (IOException e) {
}