We are running a webapplication with low number of users, about 7, but with high load of showing images. Recently we upgraded to Java 6u43 and Tomcat 6.036 and after some usage (about a week) tomcat services crashes. After a restart it can run for a few days, or just a few hours and crashes again. Even after reboot of the server it crashes again, we even saw that after 10 minutes of being active.
it is always with the same message in a hs_err_pid file in the bin folder of tomcat. See below an example of the full error message. It is always insufficient native memory and the main active thread is always JavaThread "C2 CompilerThread1", but always in a different method.
We are running on windows 2008 service pack 2, 32 bit, 2 CPU, 3GB. For tomcat xms set to 512KB and xmx to 1536KB. no other specific settings. When it crashes the Java heap is far below full usage of the 1536.
Any idea, suggestions what this could be?
(before the tomcat upgrade we used tomcat 6.014 and Java 6u02 for long time without any issue)
# # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 629784 bytes for Chunk::new # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (allocation.cpp:317), pid=5156, tid=1820 # # JRE version: 6.0_43-b01 # Java VM: Java HotSpot(TM) Server VM (20.14-b01 mixed mode windows-x86 ) --------------- T H R E A D --------------- Current thread (0x00f99800): JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=1820, stack(0x66680000,0x666f0000)] Stack: [0x66680000,0x666f0000], sp=0x666ee694, free space=441k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) [error occurred during error reporting (printing native stack), id 0xc0000005] Current CompileTask: C2:398266147 1679 ! org.apache.jasper.compiler.Compiler.isOutDated(Z)Z (485 bytes) --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x66870c00 JavaThread "http-8080-4" daemon [_thread_blocked, id=4856, stack(0x6b6a0000,0x6b710000)] 0x66870800 JavaThread "http-8080-3" daemon [_thread_blocked, id=5684, stack(0x6b170000,0x6b1e0000)] 0x66870000 JavaThread "ajp-8018-5" daemon [_thread_blocked, id=3532, stack(0x6b100000,0x6b170000)] 0x6686fc00 JavaThread "ajp-8018-4" daemon [_thread_blocked, id=1400, stack(0x6b050000,0x6b0c0000)] 0x6686f400 JavaThread "ajp-8018-3" daemon [_thread_blocked, id=2804, stack(0x6afe0000,0x6b050000)] 0x6686ec00 JavaThread "http-8080-2" daemon [_thread_blocked, id=3940, stack(0x68fd0000,0x69040000)] 0x6686e800 JavaThread "http-8080-1" daemon [_thread_blocked, id=5168, stack(0x68f60000,0x68fd0000)] 0x6686e000 JavaThread "ajp-8018-2" daemon [_thread_blocked, id=4260, stack(0x68ef0000,0x68f60000)] 0x6686dc00 JavaThread "ajp-8018-1" daemon [_thread_blocked, id=5476, stack(0x68e80000,0x68ef0000)] 0x6686d400 JavaThread "ajp-8018-Acceptor-0" daemon [_thread_in_native, id=2640, stack(0x68e10000,0x68e80000)] 0x6686d000 JavaThread "ajp-8018-CometPoller-7" daemon [_thread_blocked, id=4672, stack(0x68da0000,0x68e10000)] 0x6686c800 JavaThread "ajp-8018-CometPoller-6" daemon [_thread_blocked, id=4892, stack(0x68d30000,0x68da0000)] 0x6686c000 JavaThread "ajp-8018-CometPoller-5" daemon [_thread_blocked, id=1836, stack(0x68cc0000,0x68d30000)] 0x6686bc00 JavaThread "ajp-8018-CometPoller-4" daemon [_thread_blocked, id=5452, stack(0x68c50000,0x68cc0000)] 0x6686b400 JavaThread "ajp-8018-CometPoller-3" daemon [_thread_blocked, id=584, stack(0x68be0000,0x68c50000)] 0x66fc7000 JavaThread "ajp-8018-CometPoller-2" daemon [_thread_blocked, id=1932, stack(0x68b70000,0x68be0000)] 0x66fc6800 JavaThread "ajp-8018-CometPoller-1" daemon [_thread_blocked, id=5540, stack(0x68b00000,0x68b70000)] 0x66fc6000 JavaThread "ajp-8018-CometPoller-0" daemon [_thread_blocked, id=2728, stack(0x68a90000,0x68b00000)] 0x66fc5c00 JavaThread "ajp-8018-Poller-7" daemon [_thread_blocked, id=5104, stack(0x68a20000,0x68a90000)] 0x66fc5400 JavaThread "ajp-8018-Poller-6" daemon [_thread_blocked, id=3756, stack(0x689b0000,0x68a20000)] 0x66fc5000 JavaThread "ajp-8018-Poller-5" daemon [_thread_blocked, id=4888, stack(0x68940000,0x689b0000)] 0x66fc4800 JavaThread "ajp-8018-Poller-4" daemon [_thread_in_native, id=3924, stack(0x688d0000,0x68940000)] 0x66fc4400 JavaThread "ajp-8018-Poller-3" daemon [_thread_blocked, id=2632, stack(0x68860000,0x688d0000)] 0x66fc3c00 JavaThread "ajp-8018-Poller-2" daemon [_thread_blocked, id=3792, stack(0x687f0000,0x68860000)] 0x66fc3400 JavaThread "ajp-8018-Poller-1" daemon [_thread_blocked, id=4840, stack(0x68380000,0x683f0000)] 0x66fc3000 JavaThread "ajp-8018-Poller-0" daemon [_thread_blocked, id=5424, stack(0x68310000,0x68380000)] 0x66fc2800 JavaThread "http-8080-Acceptor-0" daemon [_thread_in_native, id=5532, stack(0x682a0000,0x68310000)] 0x66fc2400 JavaThread "http-8080-Sendfile-0" daemon [_thread_blocked, id=2560, stack(0x68230000,0x682a0000)] 0x66fc1c00 JavaThread "http-8080-CometPoller-7" daemon [_thread_blocked, id=4264, stack(0x681c0000,0x68230000)] 0x66fc1800 JavaThread "http-8080-CometPoller-6" daemon [_thread_blocked, id=5948, stack(0x68150000,0x681c0000)] 0x66fc1000 JavaThread "http-8080-CometPoller-5" daemon [_thread_blocked, id=3768, stack(0x680e0000,0x68150000)] 0x66fc0800 JavaThread "http-8080-CometPoller-4" daemon [_thread_blocked, id=5772, stack(0x68070000,0x680e0000)] 0x66fc0400 JavaThread "http-8080-CometPoller-3" daemon [_thread_blocked, id=564, stack(0x68000000,0x68070000)] 0x66fbfc00 JavaThread "http-8080-CometPoller-2" daemon [_thread_blocked, id=1192, stack(0x67f90000,0x68000000)] 0x66fbf800 JavaThread "http-8080-CometPoller-1" daemon [_thread_blocked, id=2776, stack(0x67f20000,0x67f90000)] 0x66c2f800 JavaThread "http-8080-CometPoller-0" daemon [_thread_blocked, id=1012, stack(0x67eb0000,0x67f20000)] 0x66dbe400 JavaThread "http-8080-Poller-7" daemon [_thread_blocked, id=3288, stack(0x67e40000,0x67eb0000)] 0x66d9d800 JavaThread "http-8080-Poller-6" daemon [_thread_blocked, id=3684, stack(0x67dd0000,0x67e40000)] 0x669af000 JavaThread "http-8080-Poller-5" daemon [_thread_blocked, id=4244, stack(0x67d60000,0x67dd0000)] 0x6701b000 JavaThread "http-8080-Poller-4" daemon [_thread_blocked, id=2604, stack(0x67a90000,0x67b00000)] 0x66f4f000 JavaThread "http-8080-Poller-3" daemon [_thread_blocked, id=2276, stack(0x67a20000,0x67a90000)] 0x66509000 JavaThread "http-8080-Poller-2" daemon [_thread_blocked, id=5324, stack(0x679b0000,0x67a20000)] 0x66c5a400 JavaThread "http-8080-Poller-1" daemon [_thread_blocked, id=2628, stack(0x678f0000,0x67960000)] 0x66b9a800 JavaThread "http-8080-Poller-0" daemon [_thread_blocked, id=5576, stack(0x67850000,0x678c0000)] 0x668b4000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=3152, stack(0x677e0000,0x67850000)] 0x66853000 JavaThread "GC Daemon" daemon [_thread_blocked, id=1636, stack(0x675e0000,0x67650000)] 0x00f9c800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3184, stack(0x666f0000,0x66760000)] =>0x00f99800 JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=1820, stack(0x66680000,0x666f0000)] 0x00f98400 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3932, stack(0x66610000,0x66680000)] 0x00f96c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=2612, stack(0x663a0000,0x66410000)] 0x00f95c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5128, stack(0x66330000,0x663a0000)] 0x00f68800 JavaThread "Finalizer" daemon [_thread_blocked, id=4868, stack(0x662c0000,0x66330000)] 0x00f62000 JavaThread "Reference Handler" daemon [_thread_blocked, id=5500, stack(0x66250000,0x662c0000)] 0x00e5a000 JavaThread "main" [_thread_in_native, id=2800, stack(0x00d70000,0x00de0000)] Other Threads: 0x00f5e800 VMThread [stack: 0x661e0000,0x66250000] [id=4608] 0x00f9d000 WatcherThread [stack: 0x66760000,0x667d0000] [id=3600] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap PSYoungGen total 368192K, used 215481K [0x46860000, 0x65730000, 0x65c60000) eden space 230016K, 84% used [0x46860000,0x5263eae8,0x54900000) from space 138176K, 15% used [0x5cfb0000,0x5e43f968,0x656a0000) to space 137920K, 0% used [0x54900000,0x54900000,0x5cfb0000) PSOldGen total 349568K, used 320646K [0x08060000, 0x1d5c0000, 0x46860000) object space 349568K, 91% used [0x08060000,0x1b981b60,0x1d5c0000) PSPermGen total 33664K, used 33565K [0x04060000, 0x06140000, 0x08060000) object space 33664K, 99% used [0x04060000,0x061277e8,0x06140000) Code Cache [0x00fa0000, 0x01570000, 0x03fa0000) total_blobs=1881 nmethods=1658 adapters=176 free_code_cache=44292736 largest_free_block=10112 Dynamic libraries: 0x00400000 - 0x00417000 D:\Kappa\apache-tomcat-6.0.36\bin\tomcat6.exe 0x77460000 - 0x77587000 C:\Windows\system32\ntdll.dll 0x76490000 - 0x7656c000 C:\Windows\system32\kernel32.dll 0x75ef0000 - 0x75fb6000 C:\Windows\system32\ADVAPI32.dll 0x75fc0000 - 0x76083000 C:\Windows\system32\RPCRT4.dll 0x76570000 - 0x77080000 C:\Windows\system32\SHELL32.dll 0x762d0000 - 0x7637a000 C:\Windows\system32\msvcrt.dll 0x75d30000 - 0x75d7b000 C:\Windows\system32\GDI32.dll 0x75e50000 - 0x75eed000 C:\Windows\system32\USER32.dll 0x76270000 - 0x762c9000 C:\Windows\system32\SHLWAPI.dll 0x775d0000 - 0x775ee000 C:\Windows\system32\IMM32.DLL 0x775f0000 - 0x776b8000 C:\Windows\system32\MSCTF.dll 0x77590000 - 0x77599000 C:\Windows\system32\LPK.DLL 0x75d80000 - 0x75dfd000 C:\Windows\system32\USP10.dll 0x748d0000 - 0x74a6e000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_5cb72f96088b0de0\comctl32.dll 0x6db90000 - 0x6dfea000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\server\jvm.dll 0x743e0000 - 0x74412000 C:\Windows\system32\WINMM.dll 0x77240000 - 0x77385000 C:\Windows\system32\ole32.dll 0x771b0000 - 0x7723d000 C:\Windows\system32\OLEAUT32.dll 0x74380000 - 0x743b9000 C:\Windows\system32\OLEACC.dll 0x7c340000 - 0x7c396000 C:\Windows\system32\MSVCR71.dll 0x75ae0000 - 0x75b0c000 C:\Windows\system32\apphelp.dll 0x6d870000 - 0x6d87c000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\verify.dll 0x6d3d0000 - 0x6d3ef000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\java.dll 0x6d8b0000 - 0x6d8bf000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\zip.dll 0x6d600000 - 0x6d609000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\management.dll 0x673d0000 - 0x674d7000 D:\Kappa\apache-tomcat-6.0.36\bin\tcnative-1.dll 0x75cd0000 - 0x75cd7000 C:\Windows\system32\PSAPI.DLL 0x775a0000 - 0x775cd000 C:\Windows\system32\WS2_32.dll 0x760a0000 - 0x760a6000 C:\Windows\system32\NSI.dll 0x75300000 - 0x7533b000 C:\Windows\system32\MSWSOCK.dll 0x75060000 - 0x7509b000 C:\Windows\system32\rsaenh.dll 0x6d850000 - 0x6d858000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\sunmscapi.dll 0x755e0000 - 0x756d2000 C:\Windows\system32\CRYPT32.dll 0x75740000 - 0x75752000 C:\Windows\system32\MSASN1.dll 0x75b90000 - 0x75bae000 C:\Windows\system32\USERENV.dll 0x75b70000 - 0x75b84000 C:\Windows\system32\Secur32.dll 0x74f00000 - 0x74f05000 C:\Windows\System32\wshtcpip.dll 0x6d6b0000 - 0x6d6c3000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\net.dll 0x75360000 - 0x75365000 C:\Windows\System32\wship6.dll 0x74c20000 - 0x74c2f000 C:\Windows\system32\NLAapi.dll 0x75540000 - 0x75559000 C:\Windows\system32\IPHLPAPI.DLL 0x75500000 - 0x75535000 C:\Windows\system32\dhcpcsvc.DLL 0x75780000 - 0x757ac000 C:\Windows\system32\DNSAPI.dll 0x754f0000 - 0x754f7000 C:\Windows\system32\WINNSI.DLL 0x754c0000 - 0x754e2000 C:\Windows\system32\dhcpcsvc6.DLL 0x74290000 - 0x7429f000 C:\Windows\system32\napinsp.dll 0x74270000 - 0x74278000 C:\Windows\System32\winrnr.dll 0x75e00000 - 0x75e49000 C:\Windows\system32\WLDAP32.dll 0x74280000 - 0x74286000 C:\Windows\system32\rasadhlp.dll 0x003d0000 - 0x003dd000 D:\Kappa\Java\jdk1.6.0_43\bin\ntlmauth.dll 0x75410000 - 0x75417000 C:\Windows\system32\credssp.dll 0x750d0000 - 0x75114000 C:\Windows\system32\schannel.dll 0x759a0000 - 0x75a16000 C:\Windows\system32\NETAPI32.dll 0x752c0000 - 0x752f7000 C:\Windows\system32\msv1_0.dll 0x757b0000 - 0x757c1000 C:\Windows\system32\cryptdll.dll 0x6d0b0000 - 0x6d1fc000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\awt.dll 0x707c0000 - 0x70802000 C:\Windows\system32\WINSPOOL.DRV 0x740d0000 - 0x74155000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6001.18000_none_886786f450a74a05\COMCTL32.dll 0x00920000 - 0x0092b000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\mlib_jai_util.dll 0x6b250000 - 0x6b69e000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\mlib_jai_mmx.dll 0x6d220000 - 0x6d24f000 D:\Kappa\Java\jdk1.6.0_43\jre\bin\cmm.dll VM Arguments: jvm_args: -Dcatalina.base=D:\Kappa\apache-tomcat-6.0.36 -Dcatalina.home=D:\Kappa\apache-tomcat-6.0.36 -Djava.endorsed.dirs=D:\Kappa\apache-tomcat-6.0.36\endorsed -Djava.io.tmpdir=D:\Kappa\apache-tomcat-6.0.36\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=D:\Kappa\apache-tomcat-6.0.36\conf\logging.properties -Xms512m -Xmx1500m java_command: <unknown> Launcher Type: generic Environment Variables: JAVA_HOME=D:\Kappa\Java\jdk1.6.0_43 CLASSPATH=D:\Kappa\apache-tomcat-6.0.36\lib\servlet-api.jar PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;D:\Kappa\Java\jdk1.6.0_43\bin;C:\Program Files\Windows Imaging\ USERNAME=SVC-UPSLKappa-DIA OS=Windows_NT PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 6, GenuineIntel --------------- S Y S T E M --------------- OS: Windows Server 2008 Build 6002 Service Pack 2 CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 23 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1 Memory: 4k page, physical 2095900k(292724k free), swap 4467300k(2557472k free) vm_info: Java HotSpot(TM) Server VM (20.14-b01) for windows-x86 JRE (1.6.0_43-b01), built on Mar 1 2013 02:47:24 by "java_re" with MS VC++ 7.1 (VS2003) time: Tue Oct 01 08:51:47 2013 elapsed time: 398205 seconds