MySQL/Hibernate - How do I debug a MySQL pooled connection that keeps dropping?

Question

For months, my web application ran smoothly, but for the past week or two, it keeps dropping its connection to MySQL server. I\'m not a DBA guy and have no idea how to debug thi

Answer 1

Hibernate errors are a bit abstract and sometimes it can be tricky to find the bug by the stack trace. I think that may be a problem of your application, maybe you're not closing Hibernate connections properly on some cases or your application may have a memory leak.

Have you tried to monitor the application with jconsole from the JDK?

You can set this on your Tomcat configuration console in the Java arguments (I'm assuming you're using Tomcat), to enable the jconsole

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8086
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

Then connect to a remote process for example

localhost:8086 

and watch the threads as you go thru the operations that make the application stop.

Edit

If you're not using Tomcat and you're running your application in a Windows environment you can monitorize the threads using for example Process Explorer and monitorize your application.

Answer 2

From the stack-trace you provided, I can draw a single conclusion: you are simply running out of connections.

This can be caused by long running transactions, possibly due to slow queries or improper application transaction boundaries.

I suggest you start using FlexyPool, which supports Tomcat DBCP and get a better understanding of both the connection and transaction usage. FlexyPool provides many histograms you might be interested in, like connection acquisition time and lease time.

An, just to be on the safe side, check the MySQL driver version too and see if you're running on an outdated library.

Answer 3

It seems your connection pool cannot return a free connection to Hibernate within timeout duration. This happens because your application have very long transactions or transaction dead locks. You can try following options to fix the bug.

1. change your connection pool size in following line

   <property name="hibernate.connection.pool_size">5</property>

make the pool size about 10 and test. You should keep your eye on the count of connections to your database. If it exceeds the mysql database connection limitations change max_connections of mysql server and keep testing.

2. Use an another connection pool. I recommend to use apache commons dbcp2. Maven dependencies of dbcp2 as follows.

   <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-dbcp2</artifactId> <version>2.1</version> </dependency>

Add dbcp2 into your POM then config dbcp2 with your application.

If it was the solution your application had only long transactions. Sometimes it may minimize the occurrence, and if it is still happening definitely your application have transaction dead locks. So you have to identify what are the possible problems with your code.

There are other alternative solutions such changing the waiting timeout to a higher value. But it is not good for your application performance and it doesn't make any sense for transaction dead locks. Finally you should remember to care about transaction management and database structure in your further developments for better performance of database.