How to use asyncio to download files on s3 bucket

问题

I'm using the following code to download all my files in a s3 bucket:

def main(bucket_name, destination_dir):
    bucket = boto3.resource('s3').Bucket(bucket_name)
    for obj in bucket.objects.all():
        if obj.key.endswith('/'):
            continue
        destination = '%s/%s' % (bucket_name, obj.key)
        if not os.path.exists(destination):
            os.makedirs(os.path.dirname(destination), exist_ok=True)
        bucket.download_file(obj.key, destination)

I would like to know how to make this asynchronous, if possible.

Thank u in advance.

回答1:

Aiobotocore provides asyncio support for botocore library using aiohttp. If you are willing to modify your code to use botocore instead, that would be a solution.

回答2:

Out of the box, boto3 doesn't support asyncio. There's a tracking issue opened on this that offers some workarounds; they may or may not work for your use case.

来源：https://stackoverflow.com/questions/44915400/how-to-use-asyncio-to-download-files-on-s3-bucket