Rails direct upload to Amazon S3

Question

I\'m looking to add functionality to my Rails app to upload files directly to Amazon S3. From my research the general consensus seems to be to use the s3-swf-upload-plugin.

Answer 1

Not sure about whether you can modify it easily to only upload one file at a time, but this gem works very well for me. It is based on one of Ryan Bates' Railscast:

https://github.com/waynehoover/s3_direct_upload

Answer 2

If you are using Rails 3, please check out my sample projects:

Sample project using Rails 3, Flash and MooTools-based FancyUploader to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-FancyUploader

Sample project using Rails 3, Flash/Silverlight/GoogleGears/BrowserPlus and jQuery-based Plupload to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload

By the way, you can do post-processing with Paperclip using something like this blog post describes:

http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip

Answer 3

I have adapted Heroku's direct to S3 upload solution in Rails (which uses jQuery-File-Upload and the aws-sdk gem) so uploads to S3 can be made remotely using ajax. I hope this is useful:

posts_controller.rb

before_action :set_s3_direct_post, only: [:index, :create]
before_action :delete_picture_from_s3, only: [:destroy]

class PostsController < ApplicationController

  def index
    .
    .
  end

  def create
    @post = @user.posts.build(post_params)
    if @post.save
      format.html
      format.js
    end
  end

  def destroy
    Post.find(params[:id]).destroy
  end

  private

    def set_s3_direct_post
      return S3_BUCKET.presigned_post(key: "uploads/#{SecureRandom.uuid}/${filename}", success_action_status: '201', acl: 'public-read')
    end    

    def delete_picture_from_s3
      key = params[:picture_url].split('amazonaws.com/')[1]
      S3_BUCKET.object(key).delete
      return true
      rescue => e
        # If anyone knows a good way to deal with a defunct file sitting in the bucket, please speak up.
        return true
    end

    def post_params
      params.require(:post).permit(:content, :picture_url)
    end

end

posts.html.erb

<div class="info"      data-url="<%= @s3_direct_post.url %>"
                  data-formdata="<%= (@s3_direct_post.fields.to_json) %>"
                      data-host="<%= URI.parse(@s3_direct_post.url).host %>">
</div>

The form

<%= form_for(:post, url: :posts, method: :post,
              html: { class: "post_form", id: "post_form-#{post.id}" }
            ) do |f| %>
  <%= f.text_area :content, id: "postfield-#{post.id}", class: "postText" %>
  <%= f.button( :submit, name: "Post", title: "Post" ) do %>
    <span class="glyphicon glyphicon-ok" aria-hidden="true"></span>
  <% end %>
  <span class="postuploadbutton" id="postUp-<%= post.id %>" title="Add file" >
    <span class="glyphicon glyphicon-upload" aria-hidden="true"></span>
  </span>
  <span title="Cancel file" class="noticecancelupload" id="postCancel-<%= post.id %>" >
    <span class="glyphicon glyphicon-remove-circle" aria-hidden="true"></span>
  </span>
  <%= f.file_field :picture_url, accept: 'image/jpeg,image/gif,image/png', 
               class: "notice_file_field", id: "postFile-#{post.id}" %>
<% end %>

_post.html.erb

<%= button_to post_path(
                      params: {
                        id: post.id,
                        picture_url: post.picture_url
                      }
                    ),
                    class: 'btn btn-default btn-xs blurme',
                    data: { confirm: "Delete post: are you sure?" },
                    method: :delete do %>
        <span class="glyphicon glyphicon-remove" aria-hidden="true"></span>
<% end %>

Javascript in each _post.html.erb

$(document).off('click',"#postUp-<%= post.id %>");
$(document).on('click', '#postUp-<%= post.id %>', function(e) {
  prepareUpload("#post_form-<%= post.id %>");
  $('#postFile-<%= post.id %>').trigger("click");
});

$(document).off('click',"#postCancel-<%= post.id %>");
$(document).on('click', '#postCancel-<%= post.id %>', function(e) {
  $(".appendedInput").remove(); //  $('#postFile-<% post.id %>').val(""); doesn't work for me
  $('.progBar').css('background','white').text("");
});

$(document).off('submit',"#post_form-<%= post.id %>"); // without this the form submitted multiple times in production
$(document).on('submit', '#post_form-<%= post.id %>', function(e) { // don't use $('#post_form-<%= post.id %>').submit(function() { so it doesn't bind to the #post_form (so it still works after ajax loading)
  e.preventDefault(); // prevent normal form submission
  if ( validatePostForm('<%= post.id %>') ) {
    $.ajax({
      type: 'POST',
      url:  $(this).attr('action'),
      data: $(this).serialize(),
      dataType: 'script'
    });
    $('#postCancel-<%= post.id %>').trigger("click");
  }
});

function validatePostForm(postid) {
  if ( jQuery.isBlank($('#postfield-' + postid).val()) && jQuery.isBlank($('#postFile-' + postid).val()) ) {
    alert("Write something fascinating or add a picture.");
    return false;
  } else {
    return true;
  }
}

Javascript in application.js

function prepareUpload(feckid) {
  $(feckid).find("input:file").each(function(i, elem) {
    var fileInput    = $(elem);
    var progressBar  = $("<div class='progBar'></div>");
    var barContainer = $("<div class='progress'></div>").append(progressBar);
    fileInput.after(barContainer);
    var maxFS = 10 * 1024 * 1024;

    var info             = $(".info");
    var urlnumbnuts      = info.attr("data-url");
    var formdatanumbnuts = jQuery.parseJSON(info.attr("data-formdata"));
    var hostnumbnuts     = info.attr("data-host");

    var form             = $(fileInput.parents('form:first'));

    fileInput.fileupload({
      fileInput:        fileInput,
      maxFileSize:      maxFS,
      url:              urlnumbnuts,
      type:             'POST',
      autoUpload:       true,
      formData:         formdatanumbnuts,
      paramName:        'file',
      dataType:         'XML',
      replaceFileInput: false,
      add: function (e, data) {
        $.each(data.files, function (index, file) {
          if (file.size > maxFS) {
            alert('Alas, the file exceeds the maximum file size of 10MB.');
            form[0].reset();
            return false;
          } else {
            data.submit();
            return true;
          }
        });
      },
      progressall: function (e, data) {
        var progress = parseInt(data.loaded / data.total * 100, 10);
        progressBar.css('width', progress + '%')
      },
      start: function (e) {
        progressBar.
          css('background', 'orange').
          css('display', 'block').
          css('width', '0%').
          text("Preparing...");
      },
      done: function(e, data) {
        var key   = $(data.jqXHR.responseXML).find("Key").text();
        var url   = '//' + hostnumbnuts + '/' + key;
        var input = $('<input />', { type:'hidden', class:'appendedInput', 
                     name: fileInput.attr('name'), value: url });
        form.append(input);
        progressBar.
          css('background', 'green').
          text("Ready");
      },
      fail: function(e, data) {
        progressBar.
          css("background", "red").
          css("color", "black").
          text("Failed");
      }
    });
  });
} // function prepareUpload()

create.js.erb

$(".info").attr("data-formdata",  '<%=raw @s3_direct_post.fields.to_json   %>'); // don't use .data() to set attributes 
$(".info").attr("data-url",       "<%= @s3_direct_post.url                 %>");
$(".info").attr("data-host",      "<%= URI.parse(@s3_direct_post.url).host %>");

$('.post_form')[0].reset();
$('.postText').val('');

application.js

//= require jquery-fileupload/basic

config/initializers/aws.rb

Aws.config.update({
  region: 'us-east-1',
  credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY']),
})
S3_BUCKET = Aws::S3::Resource.new.bucket(ENV['S3_BUCKET'])

Notes:

This solution is designed for multiple post forms on the index.html.erb page. This is why the @s3_direct_post information is placed inside a div of class info inside index.html.erb, rather than in each post form. This means there is only one @s3_direct_post presented on the page at any one time, irrespective of the number of forms on the page. The data inside the @s3_direct_post is only grabbed (with a call to prepareUpload()) upon clicking the file upload button. Upon submission a fresh @s3_direct_post is generated in the posts controller, and the information inside .info is updated by create.js.erb. Storing the @s3_direct_post data inside the form means many different instances of @s3_direct_post can exist at once, leading to errors with the file name generation.

You need to :set_s3_direct_post in both the posts controller index action (ready for the first upload) and the create action (ready for the second and subsequent uploads).

Normal form submission is prevented by e.preventDefault(); so it can be done 'manually' with $.ajax({. Why not just use remote: true in the form? Because in Rails, file upload is done with an HTML request and page refresh even when you try to do it remotely.

Use info.attr() rather than info.data() to set and retrieve the @s3_direct_post attributes because info.data doesn't get updated (for example see this question). This means you also have to manually parse the attribute into an object using jQuery.parseJSON() (which .data() actually does automatically).

Don't use //= require jquery-fileupload in application.js. This bug was a real ballache to identify (see here). The original Heroku solution didn't work until I changed this.

Answer 4

Try a new Gem called CarrierWaveDirect it allows you to upload files direct to S3 using a html form and easily move the image processing into a background process

Answer 5

You can use Paperclip to upload to S3 (see documentation) and to create thumbnails, although it uploads to temporary folder first, after that image processing can be applied before uploading file to S3.

As for the examples of such configuration, there are plenty of them throughout the blogosphere and on StackOverflow, e.g. this.

Answer 6

Try looking into carrierwave https://github.com/jnicklas/carrierwave (supports s3) Multi file uploads with carrierwave and uploadify http://blog.assimov.net/post/4306595758/multi-file-upload-with-uploadify-and-carrierwave-on