Uploading Large File to S3 with Ruby Fails with Out of Memory Error, How to Read and Upload in Chunks?

Question

We are uploading various files to S3 via the Ruby AWS SDK (v2) from a Windows machine. We have tested with Ruby 1.9. Our code works fine except when large files are encounte

Answer 1

The v2 AWS SDK for Ruby, aws-sdk gem, supports streaming objects directly over over the network without loading them into memory. Your example requires only a small correction to do this:

File.open(filepath, 'rb') do |file|
  resp = s3.put_object(
   :bucket => bucket,
   :key => s3key,
   :body => file
  )
end

This works because it allows the SDK to call #read on the file object passing in a small number of bytes each time. Calling #read on a Ruby IO object, such as a file, without a first argument will read the entire object into memory, returning it as a string. This is what has caused your out-of-memory errors.

That said, the aws-sdk gem provides another, more useful interface for uploading files to Amazon S3. This alternative interface automatically:

• Uses multipart APIs for large objects
• Can use multiple threads to upload parts in parallel, improving upload speed
• Computes MD5s of data client-side to for service-side data integrity checks.

A simple example:

# notice this uses Resource, not Client
s3 = Aws::S3::Resource.new(
  :access_key_id => accesskeyid,
  :secret_access_key => accesskey,
  :region => region
)

s3.bucket(bucket).object(s3key).upload_file(filepath)

This is part of the aws-sdk resource interfaces. There are quite a few helpful utilities in here. The Client class only provides basic API functionality.

Answer 2

The size limit for a bucket in .put is 5GB.

However there is "multipart" upload in s3 where you can upload the files with large size.

These links might help you: http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/MultipartUpload.html