How do I parse a CSV file located in a Amazon S3 bucket

Question

Below is the code I\'m using to parse the CSV from within the app, but I want to parse a file located in a Amazon S3 bucket. It needs to work when pushed to Heroku as well.

Answer 1

There are cases with S3, when permissions on S3 Object disallow public access. In-built Ruby functions do assume a path is publicly accessible and don't account for AWS S3 specificity.

s3 = Aws::S3::Resource.new
bucket = s3.bucket("bucket_name_here")
str = bucket.object("file_path_here").get.body.string
content = CSV.parse(str, col_sep: "\t", headers: true).map(&:to_h)

Per-line explanation using AWS SDK: Line 1. Initialize Line 2. Choose a bucket. Line 3. Choose an object and get it as a String. Line 4. Effectively CSV.parse('the string'), but I also added a options and map over it just in case it helps you.

Answer 2

You can get the csv file from S3 like this:

require 'csv'
require 'net/http'

CSV.parse(Net::HTTP.get(s3_file_url), headers: true).each do |row|
# code for processing row here
end

Answer 3

You can do it like this

CSV.new(open(path_to_s3)).each do |row|
  ...
end