Connection reset by peer when using s3, boto, django-storage for static files

Question

I\'m trying to switch to use amazon s3 to host our static files for our django project. I am using django, boto, django-storage and django-compressor. When I run collect static

Answer 1

This is issue someties occurs when you create a new bucket the first time, you have to wait for some hours or mins before you start uploading. I don't know why s3 behave like that. To prove that try creating a new bucket, point your django storage to it and u will see it show connection peer reset when u try to upload any thing from your django project, but wait for couple of hour or min try again it will work. Repeat the same step and see.

Answer 2

You have to set your bucket permissions to Authenticated Users List + Upload/Delete or you can create a specific user in IAM section of amazon and setup the access rights only for that specific user

This helped me some times ago: Setup S3 for Django

Answer 3

tl;dr

If your bucket is not in the default region, you need to tell boto which region to connect to, e.g. if your bucket is in us-west-2 you need to add the following line to settings.py:

 AWS_S3_HOST = 's3-us-west-2.amazonaws.com'

Long explanation:

It's not a permission problem and you should not set your bucket permissions to 'Authenticated users'.

This problem happens if you create your bucket in a region which is not the default one (in my case I was using us-west-2).

If you don't use the default region and you don't tell boto in which region your bucket resides, boto will connect to the default region and S3 will reply with a 307 redirect to the region where the bucket belongs.

Unfortunately, due to this bug in boto:

https://github.com/boto/boto/issues/2207

if the 307 reply arrives before boto has finished uploading the file, boto won't see the redirect and will keep uploading to the default region. Eventually S3 closes the socket resulting into a 'Connection reset by peer'.

It's a kind of race condition which depends on the size of the object being uploaded and the speed of your internet connection, which explains why it happens randomly.

There are two possible reasons why the OP stopped seeing the error after some time:

- he later created a new bucket in the default region and the problem went away by itself. 
- he started uploading only small files, which are fast enough to be fully uploaded by the time S3 replies with 307

Answer 4

I just had this issue trying to set up a second S3 bucket to use for testing/devel and what helped was deploying an older version of the application.

I have no clue why that would help, but for those of you reading this way after the fact (like me, a couple hours ago), it's worth trying to deploy a different application version.