Problems when uploading large files to Amazon S3

Question

I tried to use Amazon-SDK(Java) sample code S3TransferProgressSample.java to upload large files to Amazon-S3 storage (also posted here on AWS docs).

But

Answer 1

I think you should try Multipart API supported by AWS.

Check this out : http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html

Answer 2

I wanted to add a comment to Geoff Appleford's answer but SO wouldn't allow me to. In general his answer to use low level API works fine but even if we do now have a do-while loop the way for loop is designed there is in-built retry logic. In his code snippet the file position increases only when there is a success otherwise you are uploading the same part again.

Answer 3

The answer of Geoff Appleford works for me. However, I would add a && retryCount < MAX_RETRIES to the while loop control statement and increment of the retryCount on every exception caught inside the while.

Aviad

Answer 4

As a side note, 404 errors can be thrown if you try to do a multipart upload to a key that is already under a multipart upload.

Answer 5

Try using the low level API.

This will give you far more control when things go wrong, as they are likely to do with an 11GB file.

Requests to and from S3 do fail from time to time. With the low level API, you'll be able to retry a part of the upload if it fails.

Refactoring the example in the Amazon docs a bit:

// Step 2: Upload parts.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
    // Last part can be less than 5 MB. Adjust part size.
    partSize = Math.min(partSize, (contentLength - filePosition));

    // Create request to upload a part.
    UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(existingBucketName).withKey(keyName)
                .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                .withFileOffset(filePosition)
                .withFile(file)
                .withPartSize(partSize);

    // repeat the upload until it succeeds.
    boolean anotherPass;  
        do {
              anotherPass = false;  // assume everythings ok
              try {
                  // Upload part and add response to our list.
                  partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
              } catch (Exception e) {
                    anotherPass = true; // repeat
              }
        } while (anotherPass);

     filePosition += partSize;
}

   // Step 3: complete.
   CompleteMultipartUploadRequest compRequest = new 
                     CompleteMultipartUploadRequest(
                                existingBucketName, 
                                keyName, 
                                initResponse.getUploadId(), 
                                partETags);

   s3Client.completeMultipartUpload(compRequest);

Note: I am not a java developer so I could have messed things up syntactically, but hopefully this gets you going in the right direction. Also, you'll want to add in a 'retry counter' to prevent an endless loop if the upload repeatedly fails.