How to list all files in an s3 folder using AWS-SDK gem in ruby on rails

Question

I wanted to show a list of all files in an s3 folder so I can get all the last modified dates so I can determine what files has been changed.

I tried using objects.w

Answer 1

https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#list_objects_v2-instance_method

SDK V3 has the prefix option for client!

resp = client.list_objects_v2({ bucket: "BucketName", # required prefix: "FolderName", })

Answer 2

You can use this small piece of code for getting list of files for a specific folder.

 s3 = Aws::S3::Resource.new(region: 'ap-southeast-1', access_key_id: ENV['AWS_ACCESS_KEY_ID'], secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'] )
 data_files = s3.bucket(bucket_name).objects(prefix: 'prefix/', delimiter: 'delimiter').collect(&:key)

Answer 3

Currently I am also stuck with this problem. So far solution is to fetch all the objects and to filter them later:

data = bucket.objects(bucketname, prefix: 'Folder1')

data_without_folders = data.select { |obj| !(obj.key =~ /\/$/) }

For delimiter, you just have to pass it in bucket.objects call like:

data = bucket.objects(bucketname, prefix: 'prefix', delimiter: 'delimiter') 

If better solution is available, I will let you know.

Answer 4

Let's remember that the S3 is not a file system, so even the 'folder/' is an object.

Now if you want to get the files for a specific path, you can use the start_after. For example, if you have in your S3 bucket a list of objects like this.

pictures/
pictures/horse.jpg
pictures/dog.jpg
pictures/cat.jpg
lion.jpg

You can get all the pictures from the picture/ path by doing the following:

 s3 = Aws::S3::Resource.new(region: 'ap-southeast-1', access_key_id: ENV['AWS_ACCESS_KEY_ID'], secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'] )
 pictures = s3.bucket(bucket_name).objects(prefix: 'pictures/', delimiter: '', start_after: 'pictures/').collect(&:key)

The output is going to be only all the pictures keys without the folder/:

• pictures/horse.jpg
• pictures/dog.jpg
• pictures/cat.jpg

Answer 5

Too late answer but better than never.

You can do

s3_bucket.objects.with_prefix('folder_name').collect(&:key)

According to official documentation here