Handling MongoDB disconnect/reconnects from Node

Question

When my MongoDB connection is idle for a few minutes, the next request ends in error. From the mongo command line client, it looks like this:

> db.users.find()
Sat Jan 12 23:42:35 Socket recv() errno:54 Connection reset by peer 107.22.25.25:47207
Sat Jan 12 23:42:35 SocketException: remote: 107.22.25.25:47207 error: 9001 socket exception [1] server [107.22.25.25:47207] 
Sat Jan 12 23:42:35 DBClientCursor::init call() failed
Sat Jan 12 23:42:35 query failed : chowology.users {} to: ds047207.mongolab.com:47207
Error: error doing query: failed
Sat Jan 12 23:42:35 trying reconnect to ds047207.mongolab.com:47207
Sat Jan 12 23:42:35 reconnect ds047207.mongolab.com:47207 ok

I see the problem against sandbox instances of MongoHQ and MongoLab.

The next request goes through fine, because of the reconnect. This is a problem in my web app, because after a few minutes of inactivity, this error will come up during a web request. There are two things that surprise me:

1. That MongoDB connections are destroyed so regularly and frequently, and
2. That the driver simply raises an exception as opposed to automatically retrying after reconnecting (I'm using connect-mongo which uses mongoose which, in turn, uses node-mongodb-native).

Is this everyone else's experience? How is this supposed to be handled? I'd be surprised if app developers wrap their database operations in some retry-exception-handling nonsense.

Answer 1

You want to look at the docs for the Server object

http://mongodb.github.com/node-mongodb-native/api-generated/server.html#server

Especially the socketOptions where you can set keepAlive and the connection timeouts. By default keepalive is off and timeout is 0 or never which means the os default socket timeout is in effect (varies from os to os). Keep alive will send a packet once in awhile down the tcp socket connection to keep it alive. Sometimes firewalls are badly configured and don't send an end packet when they close a connection leaving the connection dead and in limbo which is what the monoglabs people are talking about (more often than not to be honest they are horribly configured).

Answer 2

1. Check that your computer isn't going to sleep
2. Check that your router/firewall is not killing idle connections

The first problem turned out to be my computer sleeping and dropping the network connection unknowingly. It's a new computer and I didn't realize I hadn't disabled the sleep :-P

Jared from MongoLab helped me troubleshoot this and I'm thankful for it. He said this behavior is common when going through a firewall (as mjhm suggested in his comment). So one test would be to bypass that.

Still going through my router, I get a different error after several hours idle:

db.users.find()
Sun Jan 13 14:55:02 Socket say send() errno:32 Broken pipe 107.22.25.25:47207
Error: 9001 socket exception [2] server [107.22.25.25:47207] 
Sun Jan 13 14:55:02 trying reconnect to ds047207.mongolab.com:47207
Sun Jan 13 14:55:02 reconnect ds047207.mongolab.com:47207 ok

I'll try it again from a server that doesn't go through my router/firewall.

The behavior of the driver raising the exception on the current op is expected and acceptable since a dropped connections is truly an exceptional case.

Update: Neither of these problems occur when I bypass my router, nor do they occur in my Nodejitsu instance, which I believe runs in a Joyent data center.

Answer 3

I had the same problem and I think it's because I'm accessing the internet behind a proxy server