How to upload a file to S3 without creating a temporary local file

Question

Is there any feasible way to upload a file which is generated dynamically to amazon s3 directly without first create a local file and then upload to the s3 server? I use python.

Answer 1

def upload_to_s3(url, **kwargs):
    '''
    :param url: url of image which have to upload or resize to upload
    :return: url of image stored on aws s3 bucket
    '''

    r = requests.get(url)
    if r.status_code == 200:
        # credentials stored in settings AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
        conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, host=AWS_HOST)

        # Connect to bucket and create key
        b = conn.get_bucket(AWS_Bucket_Name)
        k = b.new_key("{folder_name}/{filename}".format(**kwargs))

        k.set_contents_from_string(r.content, replace=True,
                                   headers={'Content-Type': 'application/%s' % (FILE_FORMAT)},
                                   policy='authenticated-read',
                                   reduced_redundancy=True)

        # TODO Change AWS_EXPIRY
        return k.generate_url(expires_in=AWS_EXPIRY, force_http=True)

Answer 2

I assume you're using boto. boto's Bucket.set_contents_from_file() will accept a StringIO object, and any code you have written to write data to a file should be easily adaptable to write to a StringIO object. Or if you generate a string, you can use set_contents_from_string().

Answer 3

You can try using smart_open (https://pypi.org/project/smart_open/). I used it exactly for that: writing files directly in S3.

Answer 4

Update for boto3:

aws_session = boto3.Session('my_access_key_id', 'my_secret_access_key')
s3 = aws_session.resource('s3')
s3.Bucket('my_bucket').put_object(Key='file_name.txt', Body=my_file)