Reading a file larger than 2GB into memory in Java

Question

Since ByteArrayInputStream is limited to 2GB, is there any alternate solution that allows me to store the whole contents of a 2.3GB (and possibly larger) file i

Answer 1

If you have huge quantities of memory, you really won't get any performance improvement anyway. It's only getting read in once either way, and the disk cache will ensure it gets done optimally. Just use a disk-based input stream.

Answer 2

Use NIO to read the file into a gigantic ByteBuffer, and then create a stream class that reads the ByteBuffer. There are several such floating around in open sources.

Answer 3

You can use memory writing the data compressed to a

ByteArrayOutputStream baos = new ByteArrayOutputStream
... new GZIPOutputStream(baos));

byte[] bytes = baos.toByteArray(); // < 100 MB?

ByteArrayInputStream ....

And then later wrap the input stream in a GZIPInputStream.

Still a minor slow down, but should be ideal for XML.

Answer 4

The whole point of StAX2 is that you do not need to read the file in to memory. You can just supply the source, and let the StAX StreamReader pull the data as it needs to.

What additional constraints do you have that you are not showing in your question?

If you have lots of memory, and you want to get good performance, just wrap your InputStream with a large byte buffer, and let the buffer do the buffering for you:

// 4 meg buffer on the stream
InputStream buffered = new BufferedInputStream(schemaInputStream, 1024 * 1024 * 4);

An alternative to solving this in Java is to create a RAMDisk, and to store the file on that, which would remove the problem from Java, where your basic limitation is that you can only have just less than Integer.MAX_VALUE values in a single array.