问题
I recently got the following messages in our logs followed by a JVM crash (Due to OOME). I am not sure what to make of this and would really appreciate any guidance.
2015-03-19 21:15:02,457 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:26:29,543 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19397937
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5c7d3838
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7aea62dd
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@55622ff2
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@74004a8
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
java.net.SocketOutputStream.write(SocketOutputStream.java:153)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1751)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3425)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2196)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:718)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:282)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
2015-03-19 21:56:59,137 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.java [process]:566) - Java heap space
java.lang.OutOfMemoryError: Java heap space
at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1269)
at org.exist.storage.btree.BTree$BTreeNode.access$16(BTree.java:1239)
at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:460)
at org.exist.storage.btree.BTree.scanSequential(BTree.java:413)
at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:2039)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1835)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.access$12(BTree.java:1734)
at org.exist.storage.btree.BTree.query(BTree.java:379)
at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.java:259)
at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.java:457)
at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.java:585)
at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.java:860)
at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.java:1)
at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.java:162)
at org.exist.xquery.LocationStep.getAttributes(LocationStep.java:645)
at org.exist.xquery.LocationStep.eval(LocationStep.java:434)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:446)
at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326)
at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:251)
at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:238)
at org.exist.xquery.LocationStep.eval(LocationStep.java:462)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.LetExpr.eval(LetExpr.java:142)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.BindingExpression.eval(BindingExpression.java:164)
2015-03-19 21:56:59,147 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3a9d08ca
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ecdd11
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@44ff846d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5ce5850a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@eec1d04
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6b8d4d9d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@53e9706d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@23d472cf
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4dbe4f8c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4c5e0203
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@54ac79fd
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@546e2bad
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6b13cc83
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@57e185f8
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@60357d68
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@45231180
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@3021aa73
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@6bb437ca
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@2021c9e9
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@7d53637c
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@409c2c97
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@adc5929
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@241ca71a
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@42b26866
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@636b1c33
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@b160466
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4af34669
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b53e609
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2062ebd4
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b6cfe8a
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4b7c2380
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f9be748
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78108924
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@474b002
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2ebee32f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e0fe017
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42aa175b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@637f5bac
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3a017b77
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b4f2b78
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:370)
com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2532)
com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2596)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
com.newrelic.agent.TransactionService.addTransaction(TransactionService.java:142)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1104)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1087)
com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.java:136)
com.newrelic.agent.TracerService.getTracer(TracerService.java:41)
com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.java:55)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java)
com.mysql.jdbc.DatabaseMetaData$9.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
We are using hiberanate and c3p0 with the following c3p0 configuration:
We are using the following maven artifacts for hibernate and c3p0:
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.3.6.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>4.3.6.Final</version>
</dependency>
with the following c3p0 configuration:
configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
.setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
.setProperty("hibernate.c3p0.idle_test_period", "1000")
.setProperty("hibernate.c3p0.min_size", "20")
.setProperty("hibernate.c3p0.max_size", "50")
.setProperty("hibernate.c3p0.timeout", "1800")
.setProperty("hibernate.c3p0.max_statements", "50")
The server is under very light load ~5 queries a second. (Java 8)
回答1:
So, directly the issue is that the Connection pool was trying to acquire new Connections, but the tasks attempting to acquire those tasks froze for a long period of time, so long that c3p0 decided the tasks must be deadlocked and then discarded and replaced the Thread pool. Later, the Thread pool was hung on idle connection test tasks.
Normally, "hung" tasks tend to look like the second Thread underneath the first "Pool thread stack traces:" label above: performing network IO that fails to complete. Your circumstances are odd in that two of the three threads are not stuck in IO. They have barely begun to do anything, yet they aren't live. Then you experience an OutOfMemoryError, and you get another APPARENT DEADLOCK on idle connection test tasks that also look like they mostly should be live.
Maybe your application is very close to some kind of resource limits that are causing things to run very sluggishly? Straightforwardly, you might increase the amount of memory available to this application (or modify it to have a lower memory footprint). You experience an OOME the second time the Thread pool tries to flush and recreate hung Threads, not directly provoked by that, but quite possibly caused by the growing Thread footprint. (In your logs, are there lots of these APPARENT DEADLOCKs previously? If you force a JVM Thread dump, do you see lots of com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread instances around, still hung on old, not completing tasks?
Some general comments: If your load is no more than 5-ish simultaneous queries, why such a heavy pool? why not min_size 5-ish, max 10-ish? Your max_statements setting is way too small for the number of Connections you permit in the pool. I'd omit this entirely until you get things working smoothly. Then, to get a bit better performance, you might set the more-straightforward-to-reason-about maxStatementsPerConnection parameter if you want.
Mostly you need to keep your application's footprint (memory? Threads?) well below the resources alotted to it, either by increasing resources, reducing its footprint, or fixing any issue that might exist that cause its resource footprint to increase into limits. I'd start by making the pool smaller, the available memory larger, and configuring the pool to be much smaller.
回答2:
I stumbled into the same exception and the reason was a wrong password of the db user...
c3p0 you're a funny guy
回答3:
Authentication issue for me too. I just added the domain part to the database server name and it all worked. Very misleading error.
来源：https://stackoverflow.com/questions/29158560/hibernate-apparent-deadlock-creating-emergency-threads-for-unassigned-pendin