Convert a VERY LARGE binary file into a Base64String incrementally

Question

I need help converting a VERY LARGE binary file (ZIP file) to a Base64String and back again. The files are too large to be loaded into memory all at once (they throw OutOfMe

Answer 1

You can avoid using a secondary buffer by passing offset and length to Convert.ToBase64String, like this:

private void ConvertLargeFile()
{
    using (var inputStream  = new FileStream("C:\\Users\\test\\Desktop\\my.zip", FileMode.Open, FileAccess.Read)) 
    {
        byte[] buffer = new byte[MultipleOfThree];
        int bytesRead = inputStream.Read(buffer, 0, buffer.Length);
        while(bytesRead > 0)
        {
            String base64String = Convert.ToBase64String(buffer, 0, bytesRead);
            File.AppendAllText("C:\\Users\\test\\Desktop\\Base64Zip", base64String); 
            bytesRead = inputStream.Read(buffer, 0, buffer.Length);           
        }
    }
}

The above should work, but I think Rene's answer is actually the better solution.

Answer 2

Based on the code shown in the blog from Wiktor Zychla the following code works. This same solution is indicated in the remarks section of Convert.ToBase64String as pointed out by Ivan Stoev

// using  System.Security.Cryptography

private void ConvertLargeFile()
{
    //encode 
    var filein= @"C:\Users\test\Desktop\my.zip";
    var fileout = @"C:\Users\test\Desktop\Base64Zip";
    using (FileStream fs = File.Open(fileout, FileMode.Create))
        using (var cs=new CryptoStream(fs, new ToBase64Transform(),
                                                     CryptoStreamMode.Write))

           using(var fi =File.Open(filein, FileMode.Open))
           {
               fi.CopyTo(cs);
           }
     // the zip file is now stored in base64zip    
     // and decode
     using (FileStream f64 = File.Open(fileout, FileMode.Open) )
         using (var cs=new CryptoStream(f64, new FromBase64Transform(),
                                                     CryptoStreamMode.Read ) ) 
           using(var fo =File.Open(filein +".orig", FileMode.Create))
           {
               cs.CopyTo(fo);
           }     
     // the original file is in my.zip.orig
     // use the commandlinetool 
     //  fc my.zip my.zip.orig 
     // to verify that the start file and the encoded and decoded file 
     // are the same
}

The code uses standard classes found in System.Security.Cryptography namespace and uses a CryptoStream and the FromBase64Transform and its counterpart ToBase64Transform

Answer 3

Use this code:

public void ConvertLargeFile(string source , string destination)
{
    using (FileStream inputStream = new FileStream(source, FileMode.Open, FileAccess.Read))
    { 

        int buffer_size = 30000; //or any multiple of 3

        byte[] buffer = new byte[buffer_size];
        int bytesRead = inputStream.Read(buffer, 0, buffer.Length);
        while (bytesRead > 0)
        {
            byte[] buffer2 = buffer;

            if(bytesRead < buffer_size)
            {
                buffer2 = new byte[bytesRead];
                Buffer.BlockCopy(buffer, 0, buffer2, 0, bytesRead);
            }

            string base64String = System.Convert.ToBase64String(buffer2);
            File.AppendAllText(destination, base64String);

            bytesRead = inputStream.Read(buffer, 0, buffer.Length);

        }
    }
}