Using Java to do *resumable uploads* using a *signed url* on google cloud storage

问题

Based on the doc regarding how to create an object in google-cloud-storage (see "create" method in https://googleapis.github.io/google-cloud-java/google-cloud-clients/apidocs/index.html), we should be using the blob.writer(...) method when trying to upload large files, as it presumably somehow automatically handles resumable uploads. Is this right?

However, if we wish to do resumable uploads on SIGNED urls, how does one do so in Java? (Any sample code or pointers would be very much appreciated; thus far, my research has led me to believe that it is not possible using the nicely created java libraries, and instead one needs to custom build one's own logic using "PUT" and "POST" statements in Java after one has generated the signed url. Is this the "best" way so far?)

回答1:

Regarding your first point, yes, the blob.writer(...) method does automatically handle resumable uploads. Unfortunately, this method is not callable from a signed URL, and only uploads files directly from a stream of bytes.

However, as you mentioned, it is possible to create a resumable upload from a signed URL with other methods, for instance using a PUT method seems to be a nice workaround.

What I did, was the following:

Create a signed URL with a "PUT" method. You can do so by specifying the SignUrlOption, as well I specified a service account with the required permissions in the bucket.
Use URLFetch to throw an HTTP request to this signed URL. I believe that you cannot use a curl command directly for example, and the UrlFetch API does the trick.
Add the uploadType=resumable header to the urlFetch HTTP request. See this documentation on how this works, and extra parameters and information.
I configured URLFetch to do an asynchronous call to the signed URL, as I believe it's more convenient when uploading big files.

An example code, to be used in an App Engine handler:

package com.example.storage;

import java.io.IOException;
import java.io.FileInputStream;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
import java.util.HashMap;
import java.util.Map;
import java.nio.charset.StandardCharsets;

import java.net.URL;

import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

// Cloud Storage Imports
import com.google.cloud.storage.Bucket;
import com.google.cloud.storage.BucketInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage.SignUrlOption;
import com.google.auth.oauth2.ServiceAccountCredentials;
import com.google.cloud.storage.HttpMethod;

// Url Fetch imports
import com.google.appengine.api.urlfetch.HTTPMethod;
import com.google.appengine.api.urlfetch.HTTPRequest;
import com.google.appengine.api.urlfetch.URLFetchService;
import com.google.appengine.api.urlfetch.URLFetchServiceFactory;
import com.google.appengine.api.urlfetch.HTTPHeader;

@WebServlet(name = "MainStorage", value = "/")
public class MainStorage extends HttpServlet {

        @Override
        public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
                // Bucket parameters 
                String bucketName = "MY-BUCKET-NAME";
                String blobName = "MY-BLOB-NAME";
                String keyPath = "/PATH-TO-SERVICE-ACCOUNT-KEY/key.json";

                BlobId blobId = BlobId.of(bucketName, blobName);
                Storage storage = StorageOptions.getDefaultInstance().getService();

                // Create signed URL with SignUrlOptions
                URL signedUrl = storage.signUrl(BlobInfo.newBuilder(bucketName, blobName).build(), 14, TimeUnit.DAYS,
                                                SignUrlOption.signWith(ServiceAccountCredentials.fromStream(new FileInputStream(keyPath))),
                                                SignUrlOption.httpMethod(HttpMethod.PUT));

                // Contents to upload to the Blob
                String content = "My-File-contents";

                // Build UrlFetch request
                HTTPRequest upload_request = new HTTPRequest(signedUrl, HTTPMethod.PUT);
                upload_request.setPayload(content.getBytes(StandardCharsets.UTF_8));

                // Set request to have an uploadType=resumable
                HTTPHeader set_resumable = new HTTPHeader("uploadType", "resumable");
                upload_request.setHeader(set_resumable);
                URLFetchService fetcher = URLFetchServiceFactory.getURLFetchService();

                // Do an asynchronous call to the signed URL with the contents
                fetcher.fetchAsync(upload_request);

                // Return response to App Engine handler call
                response.setContentType("text/plain");
                response.getWriter().println("Hello Storage");
        }
}

This can probably be done in a better way, but I believe it gives an idea on how such application can be made.

来源：https://stackoverflow.com/questions/54191800/using-java-to-do-resumable-uploads-using-a-signed-url-on-google-cloud-storag

标签

google-api

google-cloud-platform

google-cloud-storage