Lambda Function to write to csv and upload to S3

问题

I have a Python Script that gets the details of the unused security groups. I want that to write into a CSV file and upload to S3 Bucket.

When I test it in local machine it writes to CSV in the local machine. But when I execute that as a lambda function, it needs a place to save the CSV. So I am using s3.

import boto3
import csv

ses = boto3.client('ses')

def lambda_handler(event, context):
    with open('https://unused******- 
    1.amazonaws.com/Unused.csv', 'w') as csvfile:
        writer = csv.writer(csvfile)
        writer.writerow([
            'Account Name',
            'Region',
            'Id'
        ])
        ec2 = boto3.resource('ec2')
        sgs = list(ec2.security_groups.all())
        insts = list(ec2.instances.all())

        all_sgs = set([sg.group_id for sg in sgs])
        all_inst_sgs = set([sg['GroupId'] for inst in insts for sg in
        inst.security_groups])

        unused_sgs = all_sgs - all_inst_sgs


        for elem in unused_sgs:
            writer.writerow([
                Account_Name,
                region,
                elem
                ])

I want to write the result of "elem" into csv file and upload to S3 Bucket. Kindly advice.

回答1:

By using StringIO(), you don't need to save the csv to local and just upload the IO to S3. Try my code and let me know if something wrong because I can't test the code but it was worked for other cases.

import boto3
import csv
import io

s3 = boto3.client('s3')
ses = boto3.client('ses')

def lambda_handler(event, context):
    csvio = io.StringIO()
    writer = csv.writer(csvio)
    writer.writerow([
        'Account Name',
        'Region',
        'Id'
    ])

    ec2 = boto3.resource('ec2')
    sgs = list(ec2.security_groups.all())
    insts = list(ec2.instances.all())

    all_sgs = set([sg.group_id for sg in sgs])
    all_inst_sgs = set([sg['GroupId'] for inst in insts for sg in
    inst.security_groups])

    unused_sgs = all_sgs - all_inst_sgs

    for elem in unused_sgs:
        writer.writerow([
            Account_Name,
            region,
            elem
            ])

    s3.put_object(Body=csvio.getvalue(), ContentType='application/vnd.ms-excel', Bucket='bucket', Key='name_of.csv') 
    csvio.close()

回答2:

If the CSV file will be small, write it to the /tmp folder, then upload that file to S3. If it's large (say, larger than ~200MB) then you should probably stream it to S3.

Read the boto3 documents for the relevant S3 client methods.

回答3:

Follow jarmod advice if your csv file is small, otherwise you could use lambda to spin up a temporary ec2 instance (you can go for xlarge size for better performance) with user_data in it. The user_data will do all the csv processes on a strong and healthy ec2, though remember to terminate the instance (the termination command can be also included in the user_data) once the process is done.

来源：https://stackoverflow.com/questions/57714143/lambda-function-to-write-to-csv-and-upload-to-s3