Boto2 file upload gives ConnectionResetError

问题

I am trying to upload files to S3 bucket using code from this question: https://stackoverflow.com/a/15087468/291372. I am using boto2 (boto3 has too many dependencies). I tried many methods, but no one works for me. CORS was checked for bucket and set to allow origin from "*"

Here is my code:

# -*- coding: utf-8 -*-

import boto
import boto.s3
import sys
from boto.s3.key import Key


AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'YYYYYYYYYYYYYYYYYYYYYy'
S3_BUCKET = 'ZZZZZZZZZZZZZZZZZZZZZZ'

conn = boto.connect_s3(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
bucket = conn.get_bucket(S3_BUCKET)

testfile = "test.jpg"
print('{}: Uploading {} to Amazon S3 bucket {}'.format(datetime.now().time().isoformat(), testfile, S3_BUCKET))

def percent_cb(complete, total):
    print(datetime.now().time().isoformat(), complete, total)
#    sys.stdout.write('.')
#    sys.stdout.flush()


k = Key(bucket)
k.key = 'my test file'
k.set_contents_from_filename(testfile, cb=percent_cb, num_cb=5)

And here is result I can see in console:

19:40:19.760703: Uploading test.jpg to Amazon S3 bucket pickettagent-phase2
19:40:21.394796 0 85937
19:40:22.061834 24576 85937
19:40:24.235959 0 85937
19:40:24.973001 24576 85937
19:40:27.542148 0 85937
19:40:30.110295 0 85937
19:40:38.648783 0 85937
19:40:49.520405 0 85937
19:41:12.959745 0 85937
19:41:13.644785 24576 85937
Traceback (most recent call last):
 ......
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

Sometimes first column (complete) contains only zeroes.

After that code fails with exception ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

File I am trying to upload is about 85Kb, so it should upload in a second, but actually entire process (from start till exception) takes about 40-60 seconds.

I tried other solutions (tinys3, flask-s3, flask-upload) and nothing works!

How to upload files to S3 bucket? Maybe I missed some kind of permissions?

回答1:

Try this code, I'm using it to upload a lot of files (most of them are > 1MB):

from boto.s3.connection import S3Connection
from contextlib import contextmanager

@contextmanager
def connect_to_s3():
    conn = S3Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

    try:
        yield conn
    finally:
        conn.close()

with connect_to_s3() as conn:
    bucket = conn.get_bucket(S3_BUCKET)
    key = bucket.new_key()
    key.set_contents_from_filename(file_name, cb=percent_cb, num_cb=5)

来源：https://stackoverflow.com/questions/38123504/boto2-file-upload-gives-connectionreseterror

标签

python

amazon-s3

boto