How do I get the size of a boto3 Collection?

Question

The way I have been using is to transform the Collection into a List and query the length:

s3 = boto3.resource(\'s3\')
bucket = s3.Bucket(\'my_bucket\')
size = l         


        

Answer 1

Borrowing from a similar question, one option to retrieve the complete list of object keys from a bucket + prefix is to use recursion with the list_objects_v2 method.

This method will recursively retrieve the list of object keys, 1000 keys at a time.

Each request to list_objects_v2 uses the StartAfter argument to continue listing keys after the last key from the previous request.

import boto3

if __name__ == '__main__':

    client = boto3.client('s3',
        aws_access_key_id     = 'access_key',
        aws_secret_access_key = 'secret_key'
    )

    def get_all_object_keys(bucket, prefix, start_after = '', keys = []):
        response = client.list_objects_v2(
            Bucket     = bucket,
            Prefix     = prefix,
            StartAfter = start_after
        )

        if 'Contents' not in response:
            return keys

        key_list = response['Contents']
        last_key = key_list[-1]['Key']

        keys.extend(key_list)

        return get_all_object_keys(bucket, prefix, last_key, keys)

    object_keys = get_all_object_keys('your_bucket', 'prefix/to/files')

    print(len(object_keys))