Is it possible to copy all files from one S3 bucket to another with s3cmd?

Question

I\'m pretty happy with s3cmd, but there is one issue: How to copy all files from one S3 bucket to another? Is it even possible?

EDIT: I\'ve found a way to copy files

Answer 1

You can also use the web interface to do so:

1. Go to the source bucket in the web interface.
2. Mark the files you want to copy (use shift and mouse clicks to mark several).
3. Press Actions->Copy.
4. Go to the destination bucket.
5. Press Actions->Paste.

That's it.

Answer 2

It's actually possible. This worked for me:

import boto


AWS_ACCESS_KEY = 'Your access key'
AWS_SECRET_KEY = 'Your secret key'

conn = boto.s3.connection.S3Connection(AWS_ACCESS_KEY, AWS_SECRET_KEY)
bucket = boto.s3.bucket.Bucket(conn, SRC_BUCKET_NAME)

for item in bucket:
    # Note: here you can put also a path inside the DEST_BUCKET_NAME,
    # if you want your item to be stored inside a folder, like this:
    # bucket.copy(DEST_BUCKET_NAME, '%s/%s' % (folder_name, item.key))
    bucket.copy(DEST_BUCKET_NAME, item.key)

Answer 3

Thanks - I use a slightly modified version, where I only copy files that don't exist or are a different size, and check on the destination if the key exists in the source. I found this a bit quicker for readying the test environment:

def botoSyncPath(path):
    """
       Sync keys in specified path from source bucket to target bucket.
    """
    try:
        conn = S3Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
        srcBucket = conn.get_bucket(AWS_SRC_BUCKET)
        destBucket = conn.get_bucket(AWS_DEST_BUCKET)
        for key in srcBucket.list(path):
            destKey = destBucket.get_key(key.name)
            if not destKey or destKey.size != key.size:
                key.copy(AWS_DEST_BUCKET, key.name)

        for key in destBucket.list(path):
            srcKey = srcBucket.get_key(key.name)
            if not srcKey:
                key.delete()
    except:
        return False
    return True

Answer 4

s3cmd won't cp with only prefixes or wildcards but you can script the behavior with 's3cmd ls sourceBucket', and awk to extract the object name. Then use 's3cmd cp sourceBucket/name destBucket' to copy each object name in the list.

I use these batch files in a DOS box on Windows:

s3list.bat

s3cmd ls %1 | gawk "/s3/{ print \"\\"\"\"substr($0,index($0,\"s3://\"))\"\\"\"\"; }"

s3copy.bat

@for /F "delims=" %%s in ('s3list %1') do @s3cmd cp %%s %2

Answer 5

You can also use s3funnel which uses multi-threading:

https://github.com/neelakanta/s3funnel

example (without the access key or secret key parameters shown):

s3funnel source-bucket-name list | s3funnel dest-bucket-name copy --source-bucket source-bucket-name --threads=10