java google drive api V3 MultiPart & Resumable Upload

问题

I need help in writing multipart & resumable upload of large files (>5MB) so far I have only been able to start an multiupload upload but I don't know how to resume it either when the user pauses it or during network failures.

By "Resuming" I mean I don't know how to

1) get the total bytes already uploaded to the drive

2) how to use that value in the Content-Range Header

3) how to even pause this upload by user interaction[executeAsInputStream() Maybe?]

This is what I have done so far. I need the code to resume from where it stopped uploading even if I were to forcefully stop the application and restart it

   Drive service = GDrive.getService(); //Drive Specific Initialization Copied From QuickStart But with DriveScopes.FILES

   File fileMetadata = new File();
   fileMetadata.setName("Video.mp4"); //Video File
   fileMetadata.setMimeType("application/vnd.google-apps.video");

   java.io.File filePath = new java.io.File("E:\\large-file-60MB.mp4");//Large File Of 60 Mega Bytes
   FileContent mediaContent = new FileContent("video/mp4",filePath);

   Drive.Files.Create create=service.files().create(fileMetadata,mediaContent);

   MediaHttpUploader uploader=create.getMediaHttpUploader();
   uploader.setDirectUploadEnabled(false);                       //Use Resumable MultiPart Upload Protocol
   uploader.setChunkSize(2*MediaHttpUploader.MINIMUM_CHUNK_SIZE); //Chunks Of Bytes To Upload With Each Request

  // HttpHeaders headers=new HttpHeaders();
  // headers.put("Content-Range",?);          //This is not actual code which I used here but after reading the drive docs they talk about this header and I am not sure how or when to use it
  // uploader.setInitiationHeaders(headers);

   uploader.setProgressListener((uploading)->
   {
    switch (uploading.getUploadState())
    {
      case INITIATION_STARTED:System.out.println("Initiation has started!");
      break;
      case INITIATION_COMPLETE:System.out.println("Initiation is complete!");
      break;
      case MEDIA_IN_PROGRESS:
      System.out.println("Progress="+uploading.getProgress());
      System.out.println("Bytes="+uploading.getNumBytesUploaded());
      break;
      case MEDIA_COMPLETE:System.out.println("Upload is complete!");
    }
   });

   create.execute();

回答1:

While answering multiple questions in one answer isn't normally appropriate for Stackoverflow, it seems that these are all closely linked and so will give an overview of resumable uploads and in doing so attempt to address your three points:

How does one get total bytes already uploaded to Drive

How to use the value in the Content-Range Header

How to pause a resumable upload

From Google's documentation on Direct and Resumable Media Uploads on ther Java API Client Library documentation:

Implementation details

The main classes of interest are MediaHttpUploader and MediaHttpProgressListener.

If methods in the service-specific generated libraries contain the mediaUpload parameter in the Discovery document, then a convenience method is created for these methods that takes an InputStreamContent as a parameter.

For example, the insert method of the Drive API supports mediaUpload, and you can use the following code to upload a file:

class CustomProgressListener implements MediaHttpUploaderProgressListener {
  public void progressChanged(MediaHttpUploader uploader) throws IOException {
    switch (uploader.getUploadState()) {
      case INITIATION_STARTED:
        System.out.println("Initiation has started!");
        break;
      case INITIATION_COMPLETE:
        System.out.println("Initiation is complete!");
        break;
      case MEDIA_IN_PROGRESS:
        System.out.println(uploader.getProgress());
        break;
      case MEDIA_COMPLETE:
        System.out.println("Upload is complete!");
    }
  }
}

File mediaFile = new File("/tmp/driveFile.jpg");
InputStreamContent mediaContent =
    new InputStreamContent("image/jpeg",
        new BufferedInputStream(new FileInputStream(mediaFile)));
mediaContent.setLength(mediaFile.length());

Drive.Files.Insert request = drive.files().insert(fileMetadata, mediaContent);
request.getMediaHttpUploader().setProgressListener(new CustomProgressListener());
request.execute();

These classes however abstract away things like the location URI which is returned when creating the resumable upload, so if you want to be able to do this then you will need to follow the resumable upload initiation steps as documented here. This is all manually done however rather than directly using the Google Drive API Client Library.

To answer the first point, how you store how many bytes have been uploaded is up to you. Rather than thinking "how much is already on Drive", think "how much have I already uploaded?".

You can store this as a vairable locally if you so desire, as it will be a multiple of your chunk size (2 * MediaHttpUploader.MINIMUM_CHUNK_SIZE in your case) and should be easy to track.

The thing is, this isn't actually needed. You can just use a wildcard to indicate that the current position of your file is unknown, as per the documentation (emphasis my own):

If an upload request is terminated before a response, or if you receive a 503 Service Unavailable response, then you need to resume the interrupted upload.

To request the upload status, create an empty PUT request to the resumable session URI.

Add a Content-Range header to indicate that the current position in the file is unknown. For example, set the Content-Range to */2000000 if your total file length is 2,000,000 bytes. If you don't know the full size of the file, set the Content-Range to */*.

If you do want to keep track of the bytes, you can specify it in your Content-Range header as

Content-Range: bytes_so_far/total_bytes

Steps:

To initialise the resumable upload, you need to make a POST request to the /upload endpoint of the Drive API. You do not need to use the Drive API client library for this (and actually if you want to get the resumable session URI, you can't as the client library doesn't give you this).

Assuming you have your credential definition from:

GoogleAccountCredential credential = GoogleAccountCredential.usingOAuth2(...);

Then make the POST request containing the file metadata:

URL requestUrl = new URL("https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable");

String requestBody = "{\"name\": \"fileName\"}";

HttpURLConnection request = (HttpURLConnection) requestUrl.openConnection();

request.setRequestMethod("POST");
request.setDoInput(true);
request.setDoOutput(true);
request.setRequestProperty("Authorization", "Bearer " + credential.getToken());
request.setRequestProperty("X-Upload-Content-Type", "file/mimetype");
request.setRequestProperty("X-Upload-Content-Length", number_of_bytes_of_your_file);
request.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
request.setRequestProperty("Content-Length", String.format(Locale.ENGLISH, "%d", requestBody.getBytes().length));

OutputStream outputStream = request.getOutputStream();
outputStream.write(requestBody.getBytes());
outputStream.close();

request.connect();

The session URI - where to call to resume in case you need to - is returned in the headers of the response from the API. After connecting you can get this URI from the response:

if (request.getResponseCode() == HttpURLConnection.HTTP_OK) {
    URL sessionUri = new URL(request.getHeaderField("location"));
}

Now you have the Session URI - with this you can upload chunks of file to Drive as you please. You now need to use this URI as the upload point for successive uploads.

Remember though: A resumable session URI expires after one week.

How to Pause a Resumable Upload:

This in reality is down to how you wish to implement this. You could break out a loop for example or have a giant PAUSE THIS UPLOAD button in a GUI which toggles whether the next section of the upload continues or not.

The thing to remember is when uploading the file contents, the request made has to be done with HTTP PUT rather than POST. Following on from the previous section:

// set these variables:
long beginningOfChunk = 0;
long chunkSize = 2 * MediaHttpUploader.MINIMUM_CHUNK_SIZE;
int chunksUploaded = 0;

// Here starts the upload chunk code:
HttpURLConnection request = (HttpURLConnection) sessionUri.openConnection();

request.setRequestMethod("PUT");
request.setDoOutput(true);
// change your timeout as you desire here:
request.setConnectTimeout(30000); 
request.setRequestProperty("Content-Type", "file/mimetype");

long bytesUploadedSoFar = chunksUploaded * chunkSize;

if (beginningOfChunk + chunkSize > number_of_bytes_of_your_file) {
    chunkSize = (int) number_of_bytes_of_your_file - beginningOfChunk;
}

request.setRequestProperty("Content-Length", String.format(Locale.ENGLISH, "%d", chunkSize));
request.setRequestProperty("Content-Range", "bytes " + beginningOfChunk + "-" + (beginningOfChunk + chunkSize - 1) + "/" + number_of_bytes_of_your_file);

byte[] buffer = new byte[(int) chunksize];
FileInputStream fileInputStream = new FileInputStream(yourFile);
fileInputStream.getChannel().position(beginningOfChunk);
fileInputStream.close();

OutputStream outputStream = request.getOutputStream();
outputStream.write(buffer);
outputStream.close();
request.connect();

chunksUploaded += 1;

// End of upload chunk section

You can then call the upload chunk code on repeat; in a loop, as a function; how you like. As it is a distinct code block, you can call it as you like and therefore implement some way of pausing the upload (through breaks, sleeps, waits, etc).

Just remember: you will need to save the session URI in order to resume.

Update:

It seems that using the Drive V3 API directly to make resumable uploads isn't something yet possible. The Java Client Library documentation alludes to this when discussing when to use Drive: create vs the non service-specific libraries:

...the insert method of the Drive API supports mediaUpload, and you can use the following code to upload a file:
code block
You can also use the resumable media upload feature without the service-specific generated libraries.

Feature Request:

You can however let Google know that this is a feature that is important for the Drive API directly rather than the requirement to use the non-service specific library. Google's Issue Tracker is a place for developers to report issues and make feature requests for their development services. The page to file a Feature Request for the Drive API is here.

Note of discrepancy: what was called Drive.Files.Insert in Drive API V2 was changed to Drive.Files.Create in Drive API V3.

I know this is generally bad news, but I hope this is helpful to you!

References:

Java Quickstart | Google Drive API | Google Developers
Drive.Files.Create (Drive API v3 (Rev. 197) 1.25.0)
Create and populate folders | Google Drive API | Google Developers
Upload file data | Google Drive API | Google Developers
Files: create | Google Drive API | Google Developers
Direct and Resumable Media Uploads | API Client Library for Java