Quick way to list all files in Amazon S3 bucket?

Question

I have an amazon s3 bucket that has tens of thousands of filenames in it. What\'s the easiest way to get a text file that lists all the filenames in the bucket?

Answer 1

In javascript you can use

s3.listObjects(params, function (err, result) {});

to get all objects inside bucket. you have to pass bucket name inside params (Bucket: name).

Answer 2

For Scala developers, here it is recursive function to execute a full scan and map the contents of an AmazonS3 bucket using the official AWS SDK for Java

import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.{S3ObjectSummary, ObjectListing, GetObjectRequest}
import scala.collection.JavaConversions.{collectionAsScalaIterable => asScala}

def map[T](s3: AmazonS3Client, bucket: String, prefix: String)(f: (S3ObjectSummary) => T) = {

  def scan(acc:List[T], listing:ObjectListing): List[T] = {
    val summaries = asScala[S3ObjectSummary](listing.getObjectSummaries())
    val mapped = (for (summary <- summaries) yield f(summary)).toList

    if (!listing.isTruncated) mapped.toList
    else scan(acc ::: mapped, s3.listNextBatchOfObjects(listing))
  }

  scan(List(), s3.listObjects(bucket, prefix))
}

To invoke the above curried map() function, simply pass the already constructed (and properly initialized) AmazonS3Client object (refer to the official AWS SDK for Java API Reference), the bucket name and the prefix name in the first parameter list. Also pass the function f() you want to apply to map each object summary in the second parameter list.

For example

val keyOwnerTuples = map(s3, bucket, prefix)(s => (s.getKey, s.getOwner))

will return the full list of (key, owner) tuples in that bucket/prefix

or

map(s3, "bucket", "prefix")(s => println(s))

as you would normally approach by Monads in Functional Programming

Answer 3

After zach I would also recommend boto, but I needed to make a slight difference to his code:

conn = boto.connect_s3('access-key', 'secret'key')
bucket = conn.lookup('bucket-name')
for key in bucket:
    print key.name

Answer 4

In PHP you can get complete list of AWS-S3 objects inside specific bucket using following call

$S3 = \Aws\S3\S3Client::factory(array('region' => $region,));
$iterator = $S3->getIterator('ListObjects', array('Bucket' => $bucket));
foreach ($iterator as $obj) {
    echo $obj['Key'];
}

You can redirect output of the above code in to a file to get list of keys.

Answer 5

The EASIEST way to get a very usable text file is to download S3 Browser http://s3browser.com/ and use the Web URLs Generator to produce a list of complete link paths. It is very handy and involves about 3 clicks.

-Browse to Folder
-Select All
-Generate Urls

Best of luck to you.

Answer 6

public static Dictionary<string, DateTime> ListBucketsByCreationDate(string AccessKey, string SecretKey)  
{  

    return AWSClientFactory.CreateAmazonS3Client(AccessKey,
        SecretKey).ListBuckets().Buckets.ToDictionary(s3Bucket => s3Bucket.BucketName,
        s3Bucket => DateTime.Parse(s3Bucket.CreationDate));

}