Tomcat 8 Hangs. Requests are still coming in, but no responses are sent back

问题

I am running Tomcat 8.0.14 (recently upgraded from Tomcat 5.5) and am experiencing very inconsistent hanging of the application server.

When a hang occurs, attempting to navigate to any page causes an infinite wait
Tomcat is running as a windows service and the service is still running but must be restarted to get the application back online. I've looked at the network traffic and it appears that tomcat is receiving requests, but not generating any responses.
When a hang occurs, all of the http-nio threads are waiting to get a Connection to the database. This is probably the causes of the previous point due to all threads in a waiting mode. See below for a thread dump of one of the threads (they are all doing the same thing, so posting a whole thread dump should not be necessary). All of the nio threads appear to be waiting on the same lock (parking to wait for <7ed51011>).
We are no where near maxing out the heap space
We are logging abandoned connections and not seeing anything in the logs regarding abandoned connections.
There are absolutely no errors in any of the tomcat logs. No connection pool exhausted exceptions, out of memory exceptions, etc. NOTHING.

"http-nio-8080-exec-57" - Thread t@58985
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for <7ed51011> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
    at org.apache.tomcat.dbcp.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:582)
    at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439)
    at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:360)
    at org.apache.tomcat.dbcp.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:118)
    at org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1412)
    at xx.xx.XXXXConnection.getConnection(XXXXConnection.java:62)
    at xx.xx.XX.insert(XX.java:36)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:610)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:537)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1081)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523)
    - locked <6aa0c93c> (a org.apache.tomcat.util.net.NioChannel)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
    - locked <242c7551> (a java.util.concurrent.ThreadPoolExecutor$Worker)

We've tried the following:

We have been monitoring the application with VisualVM and other than seeing all of the threads waiting for a connection during a hang, we aren't seeing anything indicative of a problem.
We have profiled the application and are not seeing any memory leaks
We have increased the number of connections in the connection pool, but do not see any way we are hitting our maximum (500).
enabled logging of garbage collection and are only seeing collections taking fractions of a second.

I'm starting to think that this might be something in the network or ISP or something because it is very inconsistent. The application has a pretty consistent load on it at all times and will sometimes run fine for days and then crash several times in a few minutes. Maybe the network is somehow blocking the responses?

Does anyone have any ideas here? This is driving me nuts since we have nothing reporting a problem.

Update 8/31/2015

Tomcat is hanging like crazy this morning. It has hung up 7 times in an hour and a half. I've been monitoring the connections on port 8080 and it's staying somewhere around 110. The tomcat sample WAR is still available on the server during the hangs.

** Update 9/7/2015 **

Does anyone see a problem with the resource setup below?

<Resource name="jdbc/XX" auth="Container" type="javax.sql.DataSource"
        driverClassName="com.mysql.jdbc.Driver"
        maxActive="500" 
        minIdle="30" 
        maxIdle="200" 
        maxWait="20000"
        username="username" 
        password="password" 
        url="jdbc:mysql://localhost/XX?autoReconnect=true" 
    removeAbandoned="true" 
    removeAbandonedTimeout="60" 
    logAbandoned="true"/>

来源：https://stackoverflow.com/questions/32236828/tomcat-8-hangs-requests-are-still-coming-in-but-no-responses-are-sent-back

标签

java

mysql

Tomcat

hang

tomcat8