问题
The university I work at uses a Tomcat/Railo server to render ColdFusion pages. For about 6 months (before I was hired) the servers have been randomly crashing at different times, usually runnign service railo_ctl restart has fixed hte issue, however lately this hasn't been working as well.
Over the past two weeks I've narrowed the issue down, and I'm fairly sure Tomcat where the issue is coming from. I've never used Tomcat, so I'm not sure where to start. I have reviewed the Catalina.out files, and looked at the time when the servers crashed and found this error message:
Sep 18, 2013 3:17:12 PM org.apache.catalina.core.StandardServer await.
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instanc
Sep 18, 2013 3:17:12 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8888"]
Sep 18, 2013 3:17:13 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
Sep 18, 2013 3:17:14 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Sep 18, 2013 3:17:19 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
This SEVERE error prints a probably 200 times
Sep 18, 2013 11:19:26 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8888"]
Sep 18, 2013 11:19:26 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Sep 18, 2013 11:19:26 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8888"]
Sep 18, 2013 11:19:26 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8009"]
Sep 18, 2013 11:19:35 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:
/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
I also noticed that this error came up, but it doesn't seem to pop up the same time the servers crashed
Sep 18, 2013 11:06:39 AM org.apache.catalina.loader.WebappClassLoader findResourceInternal
INFO: Illegal access: this web application instance has been stopped already. Could not load META-INF/services/org.apache.xerces.xni.parser.XMLParserConfiguration. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
Sep 18, 2013 11:06:39 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.apache.xerces.parsers.XIncludeAwareParserConfiguration. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1564)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1523)
at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source)
at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.<init>(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.<init>(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.setAttribute(Unknown Source)
at railo.runtime.text.xml.XMLUtil.setAttributeEL(XMLUtil.java:270)
at railo.runtime.text.xml.XMLUtil.parse(XMLUtil.java:221)
at railo.runtime.functions.decision.IsXML.call(IsXML.java:22)
at content.feed.feedmanager_cfc$cf._2(/var/www/vhosts/my_web_app/requirements/mura/content/feed/feedManager.cfc:346)
at content.feed.feedmanager_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/feed/feedManager.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at fsu.includes.display_objects.dsp_feed_cfm$cf.call(/var/www/vhosts/my_web_app/fsu/includes/display_objects/dsp_feed.cfm:132)
at railo.runtime.PageContextImpl.doInclude(PageContextImpl.java:799)
at railo.runtime.PageContextImpl.doInclude(PageContextImpl.java:746)
at content.contentrenderer_cfc$cf._5(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc:956)
at content.contentrenderer_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:738)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at content.contentrenderer_cfc$cf._5(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc:902)
at content.contentrenderer_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:377)
at railo.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:774)
at railo.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:1495)
at content.contentrenderer_cfc$cf._call000069(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc:1113)
at content.contentrenderer_cfc$cf._5(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc:1113)
at content.contentrenderer_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:738)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at formbuilder_21.eventhandlers.contentrenderer_cfm$cf.udfCall(/var/www/vhosts/my_web_app/plugins/FormBuilder_21/eventHandlers/contentRenderer.cfm:27)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:738)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at ldapsurvey_25.eventhandlers.contentrenderer_cfm$cf.udfCall(/var/www/vhosts/my_web_app/plugins/LDAPSurvey_25/eventHandlers/contentRenderer.cfm:73)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:721)
at railo.runtime.util.VariableUtilImpl.callFunction(VariableUtilImpl.java:706)
at railo.runtime.interpreter.ref.func.UDFCall.getValue(UDFCall.java:52)
at railo.runtime.interpreter.CFMLExpressionInterpreter.interpret(CFMLExpressionInterpreter.java:187)
at railo.runtime.functions.dynamicEvaluation.Evaluate._call(Evaluate.java:76)
at railo.runtime.functions.dynamicEvaluation.Evaluate.call(Evaluate.java:69)
at railo.runtime.functions.dynamicEvaluation.Evaluate.call(Evaluate.java:22)
at content.contentrenderer_cfc$cf._8(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc:1817)
at content.contentrenderer_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:738)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at content.contentrenderer_cfc$cf._9(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc:2141)
at content.contentrenderer_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at in_contexteditor_16135.incontexteditor.eventhandler_cfc$cf.udfCall(/var/www/vhosts/my_web_app/plugins/In-ContextEditor_16/inContextEditor/EventHandler.cfc:120)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:377)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:616)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1830)
at railo.runtime.tag.Invoke.doComponent(Invoke.java:210)
at railo.runtime.tag.Invoke.doEndTag(Invoke.java:183)
at plugin.pluginmanager_cfc$cf._call000020(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginManager.cfc:1478)
at plugin.pluginmanager_cfc$cf._3(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginManager.cfc:1391)
at plugin.pluginmanager_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginManager.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:738)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at plugin.pluginmanager_cfc$cf._3(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginManager.cfc:1219)
at plugin.pluginmanager_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginManager.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at content.contentrenderer_cfc$cf._6(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc:1268)
at content.contentrenderer_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentRenderer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:377)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:616)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1830)
at railo.runtime.tag.Invoke.doComponent(Invoke.java:210)
at railo.runtime.tag.Invoke.doEndTag(Invoke.java:183)
at murascope_cfc$cf._1(/var/www/vhosts/my_web_app/requirements/mura/MuraScope.cfc:110)
at murascope_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/MuraScope.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl.onMissingMethod(ComponentImpl.java:539)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:504)
at railo.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1834)
at railo.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:769)
at railo.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:1495)
at fsu.includes.themes.fsu.templates.default_cfm$cf.call(/var/www/vhosts/my_web_app/fsu/includes/themes/fsu/templates/default.cfm:109)
at railo.runtime.PageContextImpl.doInclude(PageContextImpl.java:799)
at railo.runtime.PageContextImpl.doInclude(PageContextImpl.java:746)
at translator.standardhtmltranslator_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/Translator/standardHTMLTranslator.cfc:80)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:377)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:616)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1834)
at railo.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:769)
at railo.runtime.PageContextImpl.getFunctionWithNamedValues(PageContextImpl.java:1495)
at plugin.pluginstandardeventwrapper_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginStandardEventWrapper.cfc:121)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at handler.standardeventshandler_cfc$cf._1(/var/www/vhosts/my_web_app/requirements/mura/Handler/standardEventsHandler.cfc:60)
at handler.standardeventshandler_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/Handler/standardEventsHandler.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:377)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:616)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1830)
at railo.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:755)
at railo.runtime.util.VariableUtilImpl.callFunction(VariableUtilImpl.java:705)
at railo.runtime.interpreter.ref.func.UDFCall.getValue(UDFCall.java:52)
at railo.runtime.interpreter.CFMLExpressionInterpreter.interpret(CFMLExpressionInterpreter.java:187)
at railo.runtime.functions.dynamicEvaluation.Evaluate._call(Evaluate.java:76)
at railo.runtime.functions.dynamicEvaluation.Evaluate.call(Evaluate.java:69)
at railo.runtime.functions.dynamicEvaluation.Evaluate.call(Evaluate.java:22)
at plugin.pluginstandardeventwrapper_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginStandardEventWrapper.cfc:84)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at handler.standardeventshandler_cfc$cf._2(/var/www/vhosts/my_web_app/requirements/mura/Handler/standardEventsHandler.cfc:311)
at handler.standardeventshandler_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/Handler/standardEventsHandler.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.callWithNamedValues(UDFImpl.java:377)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:616)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.callWithNamedValues(ComponentImpl.java:1830)
at railo.runtime.util.VariableUtilImpl.callFunctionWithNamedValues(VariableUtilImpl.java:755)
at railo.runtime.util.VariableUtilImpl.callFunction(VariableUtilImpl.java:705)
at railo.runtime.interpreter.ref.func.UDFCall.getValue(UDFCall.java:52)
at railo.runtime.interpreter.CFMLExpressionInterpreter.interpret(CFMLExpressionInterpreter.java:187)
at railo.runtime.functions.dynamicEvaluation.Evaluate._call(Evaluate.java:76)
at railo.runtime.functions.dynamicEvaluation.Evaluate.call(Evaluate.java:69)
at railo.runtime.functions.dynamicEvaluation.Evaluate.call(Evaluate.java:22)
at plugin.pluginstandardeventwrapper_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/plugin/pluginStandardEventWrapper.cfc:84)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at servlet_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/servlet.cfc:85)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at mura_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/Mura.cfc:84)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at content.contentserver_cfc$cf._1(/var/www/vhosts/my_web_app/requirements/mura/content/contentServer.cfc:225)
at content.contentserver_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentServer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:738)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at content.contentserver_cfc$cf._1(/var/www/vhosts/my_web_app/requirements/mura/content/contentServer.cfc:271)
at content.contentserver_cfc$cf.udfCall(/var/www/vhosts/my_web_app/requirements/mura/content/contentServer.cfc)
at railo.runtime.type.UDFImpl.implementation(UDFImpl.java:215)
at railo.runtime.type.UDFImpl._call(UDFImpl.java:434)
at railo.runtime.type.UDFImpl.call(UDFImpl.java:384)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:615)
at railo.runtime.ComponentImpl._call(ComponentImpl.java:502)
at railo.runtime.ComponentImpl.call(ComponentImpl.java:1815)
at railo.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:733)
at railo.runtime.PageContextImpl.getFunction(PageContextImpl.java:1480)
at index_cfm$cf.call(/var/www/vhosts/my_web_app/index.cfm:53)
at railo.runtime.PageContextImpl.doInclude(PageContextImpl.java:799)
at railo.runtime.PageContextImpl.doInclude(PageContextImpl.java:751)
at railo.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:179)
at railo.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:23)
at railo.runtime.PageContextImpl.execute(PageContextImpl.java:2035)
at railo.runtime.PageContextImpl.execute(PageContextImpl.java:2002)
at railo.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:297)
at railo.loader.servlet.CFMLServlet.service(CFMLServlet.java:32)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:964)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Any thoughts or suggestions on this would be greatly appreciated.
回答1:
I believe this is the root of your problem:
SEVERE: The web application [] is still processing a request that has yet to finish
Something in your app is causing a thread to hang, and probably die. Unfortunately these kinds of errors are extremely difficult to diagnose. What I would do, if I were you, is to monitor the app with something like FusionReactor, and watch in particular for long running requests in your app. This could provide clues as to what piece of your app is causing these problems.
I would install FusionReactor in a separate instance of Tomcat so that it could monitor your app up to and after the point of it's death. Again, this could provide more clues as to what exact threads are hanging and/or dying.
FusionReactor is a commercial product but comes with a 30 day trial in which it is fully functional. If you're going to be managing this application, a license is probably going to be worthwhile, but if you can't swing that, then there are other JVM monitoring products like VisualVM and the like. They're not as targeted as FusionReactor, but they still might be able to help.
HTH
回答2:
Turn on JVM memory logging as documented here. Then feed those logs to GCViewer
That ought to give you pretty good insight into memory use. You'll see a sawtooth pattern of memory usage, but it's how low the bottom of the sawtooth pattern goes each time that tells you if you have a leak. if the troughs of the sawtooth are marching upwards, that's a sign.
You can also use kill -3 on the process to tell it to dump it's current threads to its log file. If you do that 5 seconds apart, 5 times, you can open the logs in Samurai
You're looking for threads who's stack looks the same or similar in each successive dump. They'll be the stuck ones. What's the CPU load like on the process when it crashes?
来源：https://stackoverflow.com/questions/18899147/tomcat-possible-memory-leak