List s3 buckets with its size in csv format

问题

I am trying to list the s3 buckets with its size in csv.

• Bucket Name Size
• Bucket A 2 GB
• Bucket B 10 GB

Looking for something like this...

I can list the buckets with the below code.

def main():
    with open('size.csv', 'w') as csvfile:
        writer = csv.writer(csvfile)
        writer.writerow([
            'Bucket Name',
            'Bucket Size'

        ])
        with open('accountroles.json') as ec2_file:
            ec2_data = json.load(ec2_file)
        region_list = ['us-west-1']
        for region in region_list:
            for index in range(len(ec2_data['Items'])):
                Account_Number = ec2_data['Items'][index]['Aws_Account_Number']
                Account_Name = ec2_data['Items'][index]['Acc_Name']
                ARN = ec2_data['Items'][index]['ARN']
                b = get_assume_arn_to_keys(Account_Number,Account_Name,ARN)
                #ds_client = boto3.client('s3',region_name=region,aws_access_key_id=``,aws_secret_access_key=``,aws_session_token=``)
                ds_client = boto3.client('s3',region_name=region,aws_access_key_id=b[1],aws_secret_access_key=b[2],aws_session_token=b[3])

                #s3_client = boto3.client('s3')

                bucket_list = ds_client.list_buckets()

                for bucket in bucket_list['Buckets']:
                    ************
                     ??????????
                    writer.writerow([
                        Account_Name,
                        #region,
                        bucket['Name'],
                        Bucketsize

                                   ])


main()

I can list the bucket. Please help me how to proceed with getting the sizes. I have referred few and seems the size can be got with CW metrics. Is there any way.

Help me on the script.

Edit / Update:

                bucket_list = ds_client.list_buckets()

                for bucket in bucket_list['Buckets']:

                    try:
                        lifecycle = ds_client.get_bucket_lifecycle(Bucket=bucket['Name'])
                        rules = lifecycle['Rules']
                    except:
                        rules = 'No Policy'
                    try:
                        encryption = ds_client.get_bucket_encryption(Bucket=bucket['Name'])
                        Encryptiontype = encryption['ServerSideEncryptionConfiguration']['Rules']
                    except:
                        Encryptiontype = 'Not Encrypted'

                    print(bucket['Name'], rules, Encryptiontype)

Thanks

回答1:

You can utilize list_objects(). Using that listobject you're able to get the key and size of each object using selector response['Contents'][]['size'] it holds the in bytes size of the object.

The downside of it is that in case you have too many objects in the bucket it might take time to iterate in order to get the total bucket size.

回答2:

Here's some code that will calculate the size of a bucket. I've done it as a function so you can incorporate it into your code:

import boto3

def bucket_size(bucket):
    size = 0

    s3_client = boto3.client('s3')

    paginator = s3_client.get_paginator('list_objects_v2')
    page_iterator = paginator.paginate(Bucket = bucket)

    for page in page_iterator:
        for object in page['Contents']:
            size += object['Size']

    # Return size in MB (rounded)
    return size // 1024

# Call function
size = bucket_size('my-bucket')
print(size)

I used a page iterator just in case you have more than 1000 objects in a bucket.

(In addition to the license granted under the terms of service of this site the contents of this post are licensed under MIT-0.)

来源：https://stackoverflow.com/questions/61209893/list-s3-buckets-with-its-size-in-csv-format