Upload Zip Chunks and Join them on Azure Platform

问题

I want to know that.. If I make chunks of a big Zip file and upload all chunks on Azure Cloud Storage in Container Blobs. Can i Join these chunks on Azure Platform.? for chunking i am using this code which is also generating .bat file for rejoining the chunks..

public void SplitFile(){
    int numericUpDown = 100;//in MB
    string PathToCopyChunks = "";  // path to store chunks and ( .bat  ) file
    string FilePathMakeChunks = DirectoryNameToPutScannedData; //the path of file to make chunks.
    try{
        int kbs = numericUpDown * 1024;
        int chunkSize = numericUpDown * 1024 * 1024;
        byte[] buffer = new byte[4096];
        string cmdout = "copy/b ";
        FileStream infile = File.OpenRead(FilePathMakeChunks);
        for (long i = 0; i <= infile.Length / chunkSize; i++)
        {
            string fname = Path.Combine(PathToCopyChunks, Path.Combine(PathToCopyChunks, Path.GetFileName(FilePathMakeChunks)) + "." + chunkSize + "." + i.ToString().PadLeft(4, '0') + ".part");
            string fname_x = Path.GetFileName(FilePathMakeChunks) + "." + chunkSize + "." + i.ToString().PadLeft(4, '0') + ".part";
            if (i == infile.Length / chunkSize)
                cmdout += "\"" + fname_x + "\"";
            else
                cmdout += "\"" + fname_x + "\" + ";
            FileStream outfile = File.Create(fname);
            for (int kb = 0; kb <= kbs; kb++)
            {
                int len = infile.Read(buffer, 0, 1024);
                outfile.Write(buffer, 0, len); 
            }
            outfile.Close();
        }
        cmdout += " \"" + Path.GetFileName(FilePathMakeChunks) + "\"";
        string combinerbatch = Path.Combine(PathToCopyChunks, Path.Combine(PathToCopyChunks, Path.GetFileName(DirectoryNameToPutScannedData)) + "." + chunkSize + ".combine.bat");
        File.WriteAllText(combinerbatch, cmdout);
        MessageBox.Show("Splitting Done...!");
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.ToString());
    }
}

I am uploading these chunks along with batch file in azure storage container and i want to run this batch file at my azure container to join chunks. hope this will help to understand my Question

And I am using this code for Uploading

string[] array1 = Directory.GetFiles(@"D:\Test");
string fileName = string.Empty;
foreach (string name in array1)
{
    fileName = Path.GetFileName(name);
    CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName);
    var fileStream = System.IO.File.OpenRead(name);
    blockBlob.UploadFromStream(fileStream); 
}

回答1:

There're two things:

If you're looking to zip the files in blob storage i.e. upload large files in blob storage and are expecting blob storage to zip those files there, then it is not possible. Blob storage is simple file storage.
If you have a large zip file that you want to upload in chunks and then have blob storage reassemble these chunks to create the zip file, then it is possible.

Since you didn't mention the technology you want to use, I will use Azure REST API to describe the process.

What you would need to do is split the file on the client side (from where you are uploading) in chunks. Each chunk can't be more than 4MB in size and because a block blob's maximum size can be 200GB, you can't have more than 50,000 chunks. Then you will upload these chunks using Put Block API.

Once all blocks are uploaded, you will instruct Blob storage to create the zip file using Put Block List API.

来源：https://stackoverflow.com/questions/32558511/upload-zip-chunks-and-join-them-on-azure-platform

标签

azure

storage

azure-storage

azure-storage-blobs

blobstorage