问题
during the analysis of a memory dump for a .Net4.5 WCF w3wp process, I encountered many threads identified as dead. !threads shows 68 out of 107 threads are dead which appear to be quite high. I was wondering if these threads could hold large amount of memory since the process eventually goes as high as 20GB+ and seem to never go down.
How can I inspect such threads and see the objects/memory held by these? Is it normal to have so many?
0:000> !threads
ThreadCount: 107
UnstartedThread: 0
BackgroundThread: 35
PendingThread: 0
DeadThread: 68
Hosted Runtime: no
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
7 1 16fc 0000009d253a36e0 28220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
14 2 a64 000000a1702d7560 2b220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Finalizer)
XXXX 3 0 000000a1702f9390 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
XXXX 4 0 000000a1702fa270 8038820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
16 6 21c8 000000a17031f310 102a220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
17 7 2af4 000000a170327ef0 21220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
19 9 1b50 000000a1703cccd0 1020220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
21 10 85c 000000a170416570 202b020 Preemptive 000000A0945502B8:000000A094550FD0 000000a1703360c0 0 MTA
25 11 13cc 000000a1711823f0 202b020 Preemptive 000000A094554D60:000000A094554FD0 000000a1703360c0 0 MTA
26 12 2044 000000a1711921d0 3029220 Preemptive 0000000000000000:0000000000000000 000000a1703360c0 0 MTA (Threadpool Worker)
XXXX 16 0 000000a17128a690 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
XXXX 17 0 000000a1712bd610 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
XXXX 18 0 000000a1712c5e30 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
XXXX 19 0 000000a1712c4e90 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
2 20 8a4 000000a1712c6600 20220 Preemptive 0000009E8B81C238:0000009E8B81DFD0 0000009d25385d70 0 Ukn
18 21 28f8 000000a1712c3720 20220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
22 22 bfc 000000a1712c3ef0 20220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
20 23 257c 000000a1712c5660 20220 Preemptive 000000A09457AC30:000000A09457AFD0 0000009d25385d70 0 Ukn
23 24 13e0 000000a1712c6dd0 20220 Preemptive 0000009F87F0B5C8:0000009F87F0CFD0 0000009d25385d70 0 Ukn
XXXX 26 0 000000a1713d8fb0 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
28 27 2aac 000000a1713dbe90 a029220 Preemptive 0000000000000000:0000000000000000 000000a1703360c0 0 MTA (Threadpool Completion Port)
XXXX 29 0 000000a1713dc660 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
29 30 284c 000000a1713d9f50 202b220 Preemptive 0000000000000000:0000000000000000 000000a1703360c0 0 MTA
XXXX 31 0 000000a1713da720 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 32 0 000000a1713db6c0 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
XXXX 33 0 000000a174347600 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 34 0 000000a174344720 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 35 0 000000a174345e90 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 36 0 000000a174346660 39820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
XXXX 37 0 000000a174346e30 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 38 0 000000a1743456c0 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 39 0 000000a1741b9d10 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 40 0 000000a1741bc420 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 41 0 000000a1741bcbf0 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 42 0 000000a1741ba4e0 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 43 0 000000a1741be360 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
3 44 1e94 000000a1741bd3c0 20220 Preemptive 0000009F87E511F8:0000009F87E52FD0 0000009d25385d70 0 Ukn
XXXX 45 0 000000a1741bdb90 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
35 46 12dc 000000a1741bacb0 20220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA
XXXX 47 0 000000a1741beb30 30820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
XXXX 48 0 000000a1741bf300 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 49 0 000000a171171f40 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
36 50 2bb4 000000a171173e80 202b020 Preemptive 0000000000000000:0000000000000000 000000a1703360c0 0 MTA
37 51 9e4 000000a171177530 202b020 Preemptive 000000A0945528D0:000000A094552FD0 000000a1703360c0 0 MTA
39 53 6d0 000000a171174e20 21220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
40 54 f34 000000a171172ee0 21220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
41 55 f74 000000a1711755f0 21220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
42 56 2198 000000a171174650 21220 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn
XXXX 57 0 000000a171175dc0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 60 0 000000a171176590 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 62 0 000000a171177d00 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 64 0 000000a171178ca0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 65 0 000000a1741bfad0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 70 0 000000a174344ef0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 71 0 000000a1713d9780 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 69 0 000000a171171770 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 68 0 000000a1711736b0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 67 0 000000a171172710 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 66 0 000000a171176d60 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 59 0 000000a1711784d0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 58 0 000000a1741bbc50 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 63 0 000000a1741c1240 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 61 0 000000a1741c02a0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 28 0 000000a1741c0a70 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 25 0 000000a1712c46c0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 15 0 000000a1713daef0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 14 0 000000a174347dd0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 13 0 000000a16744b400 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 52 0 000000a167448520 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 8 0 000000a16744bbd0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 72 0 000000a16744ac30 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
XXXX 73 0 000000a16744a460 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 74 0 000000a171268f50 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 75 0 000000a1712658a0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 76 0 000000a171269720 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 77 0 000000a171266070 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 78 0 000000a1712677e0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 79 0 000000a171269ef0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 80 0 000000a171266840 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 81 0 000000a17126a6c0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 82 0 000000a171267010 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 83 0 000000a17126ae90 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
XXXX 5 0 000000a171268780 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Completion Port)
43 84 dcc 000000a17126b660 8029220 Preemptive 0000009D9D1B3B88:0000009D9D1B3FD0 0000009d25385d70 0 MTA (Threadpool Completion Port)
XXXX 85 0 000000a171267fb0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 86 0 000000a17126be30 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
46 87 1e54 000000a17126c600 1029220 Preemptive 000000A094575068:000000A094576FD0 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 88 0 000000a17126cdd0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
45 89 1db8 000000a16744c3a0 1029220 Preemptive 000000A094577250:000000A094578FD0 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 90 0 000000a167448cf0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
XXXX 91 0 000000a16744cb70 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 92 0 000000a1674494c0 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 93 0 000000a16744d340 1039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Worker)
50 94 15a4 000000a16744db10 1029220 Preemptive 000000A09456AF80:000000A09456AFD0 0000009d25385d70 0 MTA (Threadpool Worker)
47 95 29c8 000000a167449c90 1029220 Preemptive 000000A094573D08:000000A094574FD0 0000009d25385d70 0 MTA (Threadpool Worker)
48 96 28c4 000000a16744e2e0 1029220 Preemptive 000000A094548ED8:000000A094548FD0 0000009d25385d70 0 MTA (Threadpool Worker)
49 97 69c 000000a16744eab0 1029220 Preemptive 0000009D9D1863F0:0000009D9D187FD0 0000009d25385d70 0 MTA (Threadpool Worker)
XXXX 98 0 000000a16744fa50 8039820 Preemptive 0000000000000000:0000000000000000 0000009d25385d70 0 Ukn (Threadpool Completion Port)
51 99 2bac 000000a16744f280 8029220 Preemptive 0000009F87F32660:0000009F87F32FD0 0000009d25385d70 0 MTA (Threadpool Completion Port)
52 101 c40 000000a174599040 1029220 Preemptive 0000009D9D178538:0000009D9D179FD0 0000009d25385d70 0 MTA (Threadpool Worker)
54 102 1e5c 000000a174598870 1029220 Preemptive 0000009F87F51578:0000009F87F52FD0 0000009d25385d70 0 MTA (Threadpool Worker)
56 103 2b68 000000a174596930 1029220 Preemptive 0000009D9D188E70:0000009D9D189FD0 0000009d25385d70 0 MTA (Threadpool Worker)
55 104 2924 000000a174595990 1029220 Preemptive 0000009D9D18C290:0000009D9D18DFD0 0000009d25385d70 0 MTA (Threadpool Worker)
53 105 2f0 000000a174599810 1029220 Preemptive 0000009E8B89EFD0:0000009E8B89FFD0 0000009d25385d70 0 MTA (Threadpool Worker)
57 106 f5c 000000a174596160 1029220 Preemptive 0000009E8B894828:0000009E8B895FD0 0000009d25385d70 0 MTA (Threadpool Worker)
58 107 20c 000000a174599fe0 1029220 Preemptive 0000009F87F53258:0000009F87F54FD0 0000009d25385d70 0 MTA (Threadpool Worker)
60 100 1f60 000000a17459a7b0 8029220 Preemptive 0000009F87F7B1A8:0000009F87F7CFD0 0000009d25385d70 0 MTA (Threadpool Completion Port)
回答1:
I was wondering if these threads could hold large amount of memory
Remember the following rule: a process provides memory, a thread consumes CPU time. The inverse is also true: a process does not run and a thread does not hold memory. If someone says "my process still runs", that's a simplification of the sentence "my process has at least one thread that still runs".
A dead thread (marked with XXXX) means that there is a .NET Thread object in memory and the "real" thread (the kernel object maintained by the operating system) is gone.
The following is an MCVE for that situation:
using System;
using System.Collections.Generic;
using System.Threading;
namespace DeadThreadExample
{
class Program
{
static List<Thread> AllThreadsIEverStarted = new List<Thread>();
static void Main()
{
for(int i=0; i<1000; i++)
{
Thread t = new Thread(DoNothing);
t.Start();
AllThreadsIEverStarted.Add(t);
t.Join();
}
Console.WriteLine("There should be 1000 dead threads now. Debug it with WinDbg and SOS !threads");
Console.ReadLine();
}
private static void DoNothing()
{
// Just nothing
}
}
}
The debugging session is:
0:006> !threads
PDB symbol for clr.dll not loaded
ThreadCount: 1002
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 1000
Hosted Runtime: no
[...]
could hold large amount of memory
0:006> !dumpheap -stat
Statistics:
MT Count TotalSize Class Name
[...]
53dde9b0 1000 20000 System.Threading.ThreadHelper
53d66bf0 1000 44000 System.Threading.ExecutionContext
53d62e10 1001 52052 System.Threading.Thread
53dad5cc 2000 64000 System.Threading.ThreadStart
So, yes, there is a "memory leak", if you call the static collection a leak. Maybe it's not a leak, because you need that information at some point in time. Once the collection is cleared, it's no longer a leak.
1000 dead threads is equivalent to ~180 kB "memory leak". I wouldn't call that "large amount". Even if you pass an object as an argument (use ParameterizedThreadStart), it seems that the property m_ThreadStartArg of the Thread object is not set, so I can hardly see how a larger amount of memory would be leaked.
If you don't like that situation, use a memory profiler and find out which GC roots still has a reference to those threads.
Is it normal to have so many?
Maybe you were just unlucky. They might all be gone with the next garbage collection.
How can I inspect such threads and see the objects/memory held by these?
Use !dumpheap -stat -type, then dumpheap -mt and then !do:
0:006> !dumpheap -stat -type Thread
Statistics:
MT Count TotalSize Class Name
[...]
53d62e10 1001 52052 System.Threading.Thread
0:006> !dumpheap -mt 53d62e10
Address MT Size
02ec247c 53d62e10 52
02ec2504 53d62e10 52
[...]
Statistics:
MT Count TotalSize Class Name
53d62e10 1001 52052 System.Threading.Thread
Total 1001 objects
0:006> !do 02ec247c
Name: System.Threading.Thread
MethodTable: 53d62e10
EEClass: 53e679a4
Size: 52(0x34) bytes
File: C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
53d6cd68 400192d 4 ....Contexts.Context 0 instance 00000000 m_Context
53d66bf0 400192e 8 ....ExecutionContext 0 instance 00000000 m_ExecutionContext
53d624e4 400192f c System.String 0 instance 00000000 m_Name
53d63c70 4001930 10 System.Delegate 0 instance 00000000 m_Delegate
53d65074 4001931 14 ...ation.CultureInfo 0 instance 00000000 m_CurrentCulture
53d65074 4001932 18 ...ation.CultureInfo 0 instance 00000000 m_CurrentUICulture
53d62734 4001933 1c System.Object 0 instance 00000000 m_ThreadStartArg
53d67b18 4001934 20 System.IntPtr 1 instance 11519f8 DONT_USE_InternalThread
53d642a8 4001935 24 System.Int32 1 instance 2 m_Priority
53d642a8 4001936 28 System.Int32 1 instance 3 m_ManagedThreadId
53d6878c 4001937 2c System.Boolean 1 instance 0 m_ExecutionContextBelongsToOuterScope
[ ... static ... ]
来源：https://stackoverflow.com/questions/58945878/large-number-of-dead-threads-in-net-memory-dump