Spring app losing connection to MySql after 8 hours. How to properly configure?

Question

I\'ve got a Spring app that I believe uses DBCP connection pooling to connect to a MySql database. I say believe because this isn\'t an area I\'m very strong in and I\'m not pos

Answer 1

My friend, DBCP does a promise he can't keep. Hehe. I've found myself with this problem and it got down to some newly firewall recently put in the middle chopping idle connections with idle time longer than X hours. So, the Db couldn't notify my client (and its socket) that the conn was going down and the socket was kept open, hence the pool couldn't know that the conn was not available. Result: first query attempt in the morning failed with timeout while the second worked as expected. Even with the validationQuery, DBCP didn't check an already valid conn (don't ask me why, I just found out that)

Solution 1? Due to the fact that it was a production environment (yeah, lots of sweat), the fast horse was to create a separate thread sending a sure-thing query to the DB using the pool every... X/4 hours. It kept the brand-new firewall/WAF from chopping my socket conn!

Solution 2? Check infrastructure. Check continuity. Check coherence in speed and mode of network interfaces (e.g full duplex, 100M). Check Db server settings (no net card saving energy hehe). And maybe keeping the probe in solution 1 working.

EDIT. testOnBorrow and validationQuery should work under normal circumstances. Imaging the pool with logical channels and a physical socket btw client and server. testOnBorrow checks if a channel is valid before giving it out to your request. It uses validationQuery to do it.

Answer 2

The short answer is it should be enough. DBCP supports testing the connection on borrowing from the connection pool (the default), but also supports test on return and test while idle.

It's also worth understanding what may be going wrong here. It sounds like something between your Tomcat server and the database is dropping the idle connection after a timeout (such as a router or firewall). The problem with this is that Tomcat thinks it still has a valid connection, tries to do some work with the connection and fails, but keeps the connection alive and returns it to the pool. Now any further attempt to talk to the database will fail if it is given the same broken connection from the pool.

I think it was Michael Nygard's excellent 'Release It!' book that described this scenario in one of his from-the-trenches stories.

You will also want to look into how MySQL cleans up dead connections as when Tomcat loses the connection after 8 hours the DB will also be unaware of the failed connection.

One final point, if you are using Tomcat 7 switch to their new connection pool as it offers better performance than DBCP.