Upload a video file by chunks

Question

Yes, it's a long question with a lot of detail... So, my question is: How can I stream an upload to Vimeo in segments?

For anyone wanting to copy and debug on their own machine: Here are the things you need:

• My code here.
• Include the Scribe library found here
• Have a valid video file (mp4) which is at least greater than 10 MB and put it in the directory C:\test.mp4 or change that code to point wherever yours is.
• That's it! Thanks for helping me out!

Big update: I've left a working API Key and Secret for Vimeo in the code here. So as long as you have a Vimeo account, all the code should work just fine for you once you've allowed the application and entered your token. Just copy the code from that link into a project on your favorite IDE and see if you can fix this with me. I'll give the bounty to whoever gives me the working code. Thanks! Oh, and don't expect to use this Key and Secret for long. Once this problem's resolved I'll delete it. :)

Overview of the problem: The problem is when I send the last chunk of bytes to Vimeo and then verify the upload, the response returns that the length of all the content is the length of only the last chunk, not all the chunks combined as it should be.

SSCCE Note: I have my entire SSCCE here. I put it somewhere else so it can be C ompilable. It is NOT very S hort (about 300 lines), but hopefully you find it to be S elf-contained, and it's certainly an E xample!). I am, however, posting the relevant portions of my code in this post.

This is how it works: When you upload a video to Vimeo via the streaming method (see Upload API documentation here for setup to get to this point), you have to give a few headers: endpoint, content-length, and content-type. The documentation says it ignores any other headers. You also give it a payload of the byte information for the file you're uploading. And then sign and send it (I have a method which will do this using scribe).

My problem: Everything works great when I just send the video in one request. My problem is in cases when I'm uploading several bigger files, the computer I'm using doesn't have enough memory to load all of that byte information and put it in the HTTP PUT request, so I have to split it up into 1 MB segments. This is where things get tricky. The documentation mentions that it's possible to "resume" uploads, so I'm trying to do that with my code, but it's not working quite right. Below, you'll see the code for sending the video. Remember my SSCCE is here.

Things I've tried: I'm thinking it has something to do with the Content-Range header... So here are the things I've tried in changing what the Content-Range header says...

• Not adding content range header to the first chunk

• Adding a prefix to the content range header (each with a combination of the previous header):

  • "bytes"
  • "bytes " (throws connection error, see the very bottom for the error) --> It appears in the documentation that this is what they're looking for, but I'm pretty sure there are typos in the documentation because they have the content-range header on their "resume" example as: 1001-339108/339108 when it should be 1001-339107/339108. So... Yeah...
  • "bytes%20"
  • "bytes:"
  • "bytes: "
  • "bytes="
  • "bytes= "

• Not adding anything as a prefix to the content range header

Here's the code:

/**
* Send the video data
*
* @return whether the video successfully sent
*/
private static boolean sendVideo(String endpoint, File file) throws FileNotFoundException, IOException {
  // Setup File
  long contentLength = file.length();
  String contentLengthString = Long.toString(contentLength);
  FileInputStream is = new FileInputStream(file);
  int bufferSize = 10485760; // 10 MB = 10485760 bytes
  byte[] bytesPortion = new byte[bufferSize];
  int byteNumber = 0;
  int maxAttempts = 1;
  while (is.read(bytesPortion, 0, bufferSize) != -1) {
    String contentRange = Integer.toString(byteNumber);
    long bytesLeft = contentLength - byteNumber;
    System.out.println(newline + newline + "Bytes Left: " + bytesLeft);
    if (bytesLeft < bufferSize) {
      //copy the bytesPortion array into a smaller array containing only the remaining bytes
      bytesPortion = Arrays.copyOf(bytesPortion, (int) bytesLeft);
      //This just makes it so it doesn't throw an IndexOutOfBounds exception on the next while iteration. It shouldn't get past another iteration
      bufferSize = (int) bytesLeft;
    }
    byteNumber += bytesPortion.length;
    contentRange += "-" + (byteNumber - 1) + "/" + contentLengthString;
    int attempts = 0;
    boolean success = false;
    while (attempts < maxAttempts && !success) {
      int bytesOnServer = sendVideoBytes("Test video", endpoint, contentLengthString, "video/mp4", contentRange, bytesPortion, first);
      if (bytesOnServer == byteNumber) {
        success = true;
      } else {
        System.out.println(bytesOnServer + " != " + byteNumber);
        System.out.println("Success is not true!");
      }
      attempts++;
    }
    first = true;
    if (!success) {
      return false;
    }
  }
  return true;
}

/**
* Sends the given bytes to the given endpoint
*
* @return the last byte on the server (from verifyUpload(endpoint))
*/
private static int sendVideoBytes(String videoTitle, String endpoint, String contentLength, String fileType, String contentRange, byte[] fileBytes, boolean addContentRange) throws FileNotFoundException, IOException {
  OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint);
  request.addHeader("Content-Length", contentLength);
  request.addHeader("Content-Type", fileType);
  if (addContentRange) {
    request.addHeader("Content-Range", contentRangeHeaderPrefix + contentRange);
  }
  request.addPayload(fileBytes);
  Response response = signAndSendToVimeo(request, "sendVideo on " + videoTitle, false);
  if (response.getCode() != 200 && !response.isSuccessful()) {
    return -1;
  }
  return verifyUpload(endpoint);
}

/**
* Verifies the upload and returns whether it's successful
*
* @param endpoint to verify upload to
* @return the last byte on the server
*/
public static int verifyUpload(String endpoint) {
  // Verify the upload
  OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint);
  request.addHeader("Content-Length", "0");
  request.addHeader("Content-Range", "bytes */*");
  Response response = signAndSendToVimeo(request, "verifyUpload to " + endpoint, true);
  if (response.getCode() != 308 || !response.isSuccessful()) {
    return -1;
  }
  String range = response.getHeader("Range");
  //range = "bytes=0-10485759"
  return Integer.parseInt(range.substring(range.lastIndexOf("-") + 1)) + 1;
  //The + 1 at the end is because Vimeo gives you 0-whatever byte where 0 = the first byte
}

Here's the signAndSendToVimeo method:

/**
* Signs the request and sends it. Returns the response.
*
* @param service
* @param accessToken
* @param request
* @return response
*/
public static Response signAndSendToVimeo(OAuthRequest request, String description, boolean printBody) throws org.scribe.exceptions.OAuthException {
  System.out.println(newline + newline
          + "Signing " + description + " request:"
          + ((printBody && !request.getBodyContents().isEmpty()) ? newline + "\tBody Contents:" + request.getBodyContents() : "")
          + ((!request.getHeaders().isEmpty()) ? newline + "\tHeaders: " + request.getHeaders() : ""));
  service.signRequest(accessToken, request);
  printRequest(request, description);
  Response response = request.send();
  printResponse(response, description, printBody);
  return response;
}

And here's some (an example... All of the output can be found here) of the output from the printRequest and printResponse methods: NOTE This output changes depending on what the contentRangeHeaderPrefix is set to and the first boolean is set to (which specifies whether or not to include the Content-Range header on the first chunk).

We're sending the video for upload!


Bytes Left: 15125120


Signing sendVideo on Test video request:
    Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%200-10485759/15125120}

sendVideo on Test video >>> Request
Headers: {Authorization=OAuth oauth_signature="zUdkaaoJyvz%2Bt6zoMvAFvX0DRkc%3D", oauth_version="1.0", oauth_nonce="340477132", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336004", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 0-10485759/15125120}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d

sendVideo on Test video >>> Response
Code: 200
Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}


Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request:
    Headers: {Content-Length=0, Content-Range=bytes */*}

verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Request
Headers: {Authorization=OAuth oauth_signature="FQg8HJe84nrUTdyvMJGM37dpNpI%3D", oauth_version="1.0", oauth_nonce="298157825", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=0, Content-Range=bytes */*}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d

verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Response
Code: 308
Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-10485759, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
Body: 


Bytes Left: 4639360


Signing sendVideo on Test video request:
    Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 10485760-15125119/15125120}

sendVideo on Test video >>> Request
Headers: {Authorization=OAuth oauth_signature="qspQBu42HVhQ7sDpzKGeu3%2Bn8tM%3D", oauth_version="1.0", oauth_nonce="183131870", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%2010485760-15125119/15125120}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d

sendVideo on Test video >>> Response
Code: 200
Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}


Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request:
    Headers: {Content-Length=0, Content-Range=bytes */*}

verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Request
Headers: {Authorization=OAuth oauth_signature="IdhhhBryzCa5eYqSPKAQfnVFpIg%3D", oauth_version="1.0", oauth_nonce="442087608", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336020", Content-Length=0, Content-Range=bytes */*}
Verb: PUT
Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d

4639359 != 15125120
verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d >>> Response
Success is not true!
Code: 308
Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-4639359, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0}
Body: 

Then the code goes on to complete the upload and set video information (you can see that in my full code).

Edit 2: Tried removing the "%20" from the content-range and received this error making connection. I must use either "bytes%20" or not add "bytes" at all...

Exception in thread "main" org.scribe.exceptions.OAuthException: Problems while creating connection.
    at org.scribe.model.Request.send(Request.java:70)
    at org.scribe.model.OAuthRequest.send(OAuthRequest.java:12)
    at autouploadermodel.VimeoTest.signAndSendToVimeo(VimeoTest.java:282)
    at autouploadermodel.VimeoTest.sendVideoBytes(VimeoTest.java:130)
    at autouploadermodel.VimeoTest.sendVideo(VimeoTest.java:105)
    at autouploadermodel.VimeoTest.main(VimeoTest.java:62)
Caused by: java.io.IOException: Error writing to server
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:622)
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:634)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1317)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at org.scribe.model.Response.<init>(Response.java:28)
    at org.scribe.model.Request.doSend(Request.java:110)
    at org.scribe.model.Request.send(Request.java:62)
    ... 5 more
Java Result: 1

Edit 1: Updated the code and output. Still need help!

Answer 1

I think your problem could simply be the result of this line:

request.addHeader("Content-Range", "bytes%20" + contentRange);

Try and replace "bytes%20" by simply "bytes "

In your output you see the corresponding header has incorrect content:

Headers: {
    Content-Length=15125120,
    Content-Type=video/mp4,
    Content-Range=bytes%200-10485759/15125120     <-- INCORRECT
}

On the topic of Content-Range...

You're right that an example final block of content should have a range like 14680064-15125119/15125120. That's part of the HTTP 1.1 spec.

Answer 2

Here

 String contentRange = Integer.toString(byteNumber + 1);

you start from 1 and not from 0 at the first iteration.

Here

 request.addHeader("Content-Length", contentLength);

you put the entire file content length and not the length of the current chunk.

Answer 3

The vimeo API page says: "The final step is to call vimeo.videos.upload.complete to queue up the video for transcoding. This call will return the video_id, which you can then use in other calls (to set the title, description, privacy, etc.). If you do not call this method, the video will not be processed."

I added this bit of code to the end and got it to work:

    request = new OAuthRequest(Verb.PUT, "http://vimeo.com/api/rest/v2");
    request.addQuerystringParameter("method", "vimeo.videos.upload.complete");
    request.addQuerystringParameter("filename", video.getName());
    request.addQuerystringParameter("ticket_id", ticket);
    service.signRequest(token, request);        

    response = request.send();