Out of memory when encoding file to base64

Question

Using Base64 from Apache commons

public byte[] encode(File file) throws FileNotFoundException, IOException {
        byte[] encoded;
        try (FileInputSt         


        

Answer 1

1. You are not reading the whole file, just the first few kb. The read method returns how many bytes were actually read. You should call read in a loop until it returns -1 to be sure that you have read everything.

2. The file is too big for both it and its base64 encoding to fit in memory. Either

   • process the file in smaller pieces or

   • increase the memory available to the JVM with the -Xmx switch, e.g.

     java -Xmx1024M YourProgram
     

Answer 2

You cannot just load the whole file into memory, like here:

byte fileContent[] = new byte[(int) file.length()];
fin.read(fileContent);

Instead load the file chunk by chunk and encode it in parts. Base64 is a simple encoding, it is enough to load 3 bytes and encode them at a time (this will produce 4 bytes after encoding). For performance reasons consider loading multiples of 3 bytes, e.g. 3000 bytes - should be just fine. Also consider buffering input file.

An example:

byte fileContent[] = new byte[3000];
try (FileInputStream fin = new FileInputStream(file)) {
    while(fin.read(fileContent) >= 0) {
         Base64.encodeBase64(fileContent);
    }
}

Note that you cannot simply append results of Base64.encodeBase64() to encoded bbyte array. Actually, it is not loading the file but encoding it to Base64 causing the out-of-memory problem. This is understandable because Base64 version is bigger (and you already have a file occupying a lot of memory).

Consider changing your method to:

public void encode(File file, OutputStream base64OutputStream)

and sending Base64-encoded data directly to the base64OutputStream rather than returning it.

UPDATE: Thanks to @StephenC I developed much easier version:

public void encode(File file, OutputStream base64OutputStream) {
  InputStream is = new FileInputStream(file);
  OutputStream out = new Base64OutputStream(base64OutputStream)
  IOUtils.copy(is, out);
  is.close();
  out.close();
}

It uses Base64OutputStream that translates input to Base64 on-the-fly and IOUtils class from Apache Commons IO.

Note: you must close the FileInputStream and Base64OutputStream explicitly to print = if required but buffering is handled by IOUtils.copy().