Can we copy the files and folders recursively between aws s3 buckets using boto3 Python?

Question

Is it possible to copy all the files in one source bucket to other target bucket using boto3. And source bucket doesn\'t have regular folder structure.

Sourc         


        

Answer 1

S3 store object, it doesn't store folder, even '/' or '\' is part of the object key name. You just need to manipulate the key name and copy the data over.

import boto3
old_bucket_name = 'SRC'
old_prefix = 'A/B/C/'
new_bucket_name = 'TGT'
new_prefix = 'L/M/N/'
s3 = boto3.resource('s3')
old_bucket = s3.Bucket(old_bucket_name)
new_bucket = s3.Bucket(new_bucket_name)

for obj in old_bucket.objects.filter(Prefix=old_prefix):
    old_source = { 'Bucket': old_bucket_name,
                   'Key': obj.key}
    # replace the prefix
    new_key = obj.key.replace(old_prefix, new_prefix, 1)
    new_obj = new_bucket.Object(new_key)
    new_obj.copy(old_source)

Optimized technique of defining new_key suggested by zvikico:

new_key = new_prefix + obj.key[len(old_prefix):]