How to upload a file to S3 without creating a temporary local file

Question

Is there any feasible way to upload a file which is generated dynamically to amazon s3 directly without first create a local file and then upload to the s3 server? I use python.

Answer 1

I am having a similar issue, was wondering if there was a final answer, because with my code below , the "starwars.json" keeps on saving locally but I just want to push through each looped .json file into S3 and have no file stored locally.

for key, value in star_wars_actors.items():

response = requests.get('http:starwarsapi/' + value)



data = response.json()


with open("starwars.json", "w+") as d:
    json.dump(data, d, ensure_ascii=False, indent=4)



s3.upload_file('starwars.json', 'test-bucket',
               '%s/%s' % ('test', str(key) + '.json'))

Answer 2

I had a dict object which I wanted to store as a json file on S3, without creating a local file. The below code worked for me:

from smart_open import smart_open

with smart_open('s3://access-key:secret-key@bucket-name/file.json', 'wb') as fout:
    fout.write(json.dumps(dict_object).encode('utf8'))

Answer 3

Here is an example downloading an image (using requests library) and uploading it to s3, without writing to a local file:

import boto
from boto.s3.key import Key
import requests

#setup the bucket
c = boto.connect_s3(your_s3_key, your_s3_key_secret)
b = c.get_bucket(bucket, validate=False)

#download the file
url = "http://en.wikipedia.org/static/images/project-logos/enwiki.png"
r = requests.get(url)
if r.status_code == 200:
    #upload the file
    k = Key(b)
    k.key = "image1.png"
    k.content_type = r.headers['content-type']
    k.set_contents_from_string(r.content)

Answer 4

The boto library's Key object has several methods you might be interested in:

• send_file
• set_contents_from_file
• set_contents_from_string
• set_contents_from_stream

For an example of using set_contents_from_string, see Storing Data section of the boto documentation, pasted here for completeness:

>>> from boto.s3.key import Key
>>> k = Key(bucket)
>>> k.key = 'foobar'
>>> k.set_contents_from_string('This is a test of S3')

Answer 5

Given that encryption at rest is a much desired data standard now, smart_open does not support this afaik

Answer 6

You could use BytesIO from the Python standard library.

from io import BytesIO
bytesIO = BytesIO()
bytesIO.write('whee')
bytesIO.seek(0)
s3_file.set_contents_from_file(bytesIO)