104, 'Connection reset by peer' socket error, or When does closing a socket result in a RST rather than FIN?

Question

We\'re developing a Python web service and a client web site in parallel. When we make an HTTP request from the client to the service, one call consistently raises a socket

Answer 1

I had the same issue however with doing an upload of a very large file using a python-requests client posting to a nginx+uwsgi backend.

What ended up being the cause was the the backend had a cap on the max file size for uploads lower than what the client was trying to send.

The error never showed up in our uwsgi logs since this limit was actually one imposed by nginx.

Upping the limit in nginx removed the error.

Answer 2

I've had this problem. See The Python "Connection Reset By Peer" Problem.

You have (most likely) run afoul of small timing issues based on the Python Global Interpreter Lock.

You can (sometimes) correct this with a time.sleep(0.01) placed strategically.

"Where?" you ask. Beats me. The idea is to provide some better thread concurrency in and around the client requests. Try putting it just before you make the request so that the GIL is reset and the Python interpreter can clear out any pending threads.

Answer 3

Normally, you'd get an RST if you do a close which doesn't linger (i.e. in which data can be discarded by the stack if it hasn't been sent and ACK'd) and a normal FIN if you allow the close to linger (i.e. the close waits for the data in transit to be ACK'd).

Perhaps all you need to do is set your socket to linger so that you remove the race condition between a non lingering close done on the socket and the ACKs arriving?

Answer 4

Don't use wsgiref for production. Use Apache and mod_wsgi, or something else.

We continue to see these connection resets, sometimes frequently, with wsgiref (the backend used by the werkzeug test server, and possibly others like the Django test server). Our solution was to log the error, retry the call in a loop, and give up after ten failures. httplib2 tries twice, but we needed a few more. They seem to come in bunches as well - adding a 1 second sleep might clear the issue.

We've never seen a connection reset when running through Apache and mod_wsgi. I don't know what they do differently, (maybe they just mask them), but they don't appear.

When we asked the local dev community for help, someone confirmed that they see a lot of connection resets with wsgiref that go away on the production server. There's a bug there, but it is going to be hard to find it.