How to use WinDbg to analyze the crash dump for VC++ application?

问题

How do I use WinDbg for analyzing a dump file?

回答1:

Here are some general steps that will get you on your way:

First, you must change your compiler's settings so that it creates PDB files, even for release builds. Later versions of the Visual C++ compiler do this by default, but in many versions of Visual C++ you must do this yourself. Create program database files, and then keep an archive of those files along with each build of your application. It is critical that every build of your applications has its own set of PDBs. You can't just reuse the same ones you made with build 10 to examining the dumps generated by build 15, for example. Over the life of your project, you will end up with a ton of PDBs, so be prepared for that.

Next, you need to be able to identify the exact version of your application which generated the dump file. If you are creating your own MiniDumps (by calling MiniDumpWriteDump() for example), probably the easiest way to do this is to simply make part of the filename of the MiniDump the complete version number of your application. You'll need to have a reasonable version numbering scheme in place for this to work. In my shop, we increment the build number across all branches by one every time the autobuilder creates a build.

Now that you have received the dump file from the customer, you know the precise version of the application that created the dump, and you have found the PDB files for this build.

Now you need to go through your source control's history and find the source code for this exact version of the software. The best way to do this is to apply 'labels' to your branches every time you make a build. Set the value of the label to the exact version number, and it becomes easy to find in the history.

You're almost ready to fire up WinDbg/Visual C++:

Get the complete source tree for that version of your application. Put it in a separate place on your hard drive, say c:\app_build_1.0.100 for application version 1.0 build #100.
Get the binaries for that exact version of your application and put them somewhere on your hard drive. It might be easiest simply to install that version of your application to get the binaries.
Put the PDB files in the same location as the binaries in step 2.

Now you have two options for viewing the dump file. You can use Visual Studio or WinDbg. Using Visual Studio is easier, but WinDbg is much more powerful. Most of the time the functionality in Visual Studio will suffice.

To use Visual Studio, all you have to do is open the dump file like it is a project. Once opened, "run" the dump file (F5 by default) and if all the paths are set correctly it will take you right to the code that crashed, give you a call stack, etc.

To use WinDbg, you have to jump through a couple of hoops:

Start WinDbg
Open the dump file. (Ctrl + D by default)
Tell WinDbg to go get the correct MicroSoft symbol files. Type .symfix. This may take a few moments as it will pull a ton of stuff down from the Internet.
Tell WinDbg where the symbols (PDB files) are. Type .sympath+ c:\pdblocation, substituting wherever you put the PDB files for the pathname. Make sure you get the plus sign in there with no whitespace between .sympath and the + sign or else you'll screw up step 3.
Tell WinDbg where the source code is. Type .srcpath c:\app_build_1.0.100 substituting the path where you got code from source control for this version of the software.
Tell WinDbg to analyze the dump file. Type !analyze -v

After a few moments, if everything is configured correctly, WinDbg will take you right to the location of your crash. At this point you have a million options for digging deep into your application's memory space, the state of critical sections, windows, etc. But that is way beyond the scope of this post.

Good luck!

回答2:

(see the "Dump" sections below)

Basic Tutorials and Demonstrations of Using WinDbg

Installing and Configuring WinDbg (Windows Debug Tools)
Mike Taulty - A word for WinDBG
WinDbg Tutorials
Windows Debuggers: Part 1: A WinDbg Tutorial

Different Ways to "Start"/Attach WinDBG

Start Debugging with Windbg (includes how to debug an .msi)
How to debug a Windows service
Setting up Windows Debugging

Workspaces

Understanding how Workspaces work...

Pimp up your debugger: Creating a custom workspace for windbg debugging
Uncovering How Workspaces Work in WinDbg

Cmdtree

A "cmdtree" allows you to define a "menu" of debugger commands for easy access to frequently used commands without having to remember the terse command names.

You don't have to put all the command definitions into the same cmdtree text file....you can keep them separate and load multiple ones if you wish (they then get their own window).

Amazing helper .cmdtree
How do I make a cmdtree window dock at startup in WinDBG
Making it easier to debug .net dumps in windbg using .cmdtree
Microshaoft Cmdtree
Special Command—Execute Commands from a Customized User Interface with .cmdtree

Startup Script

You can use the -c option on the command line to automatically run a WinDBG script when you start WinDBG.

Gives opportunity to turn on DML (Debugger markup language) mode, load particular extensions, set .NET exception breakpoints, set kernel flags (e.g. when kernel debugging you might need to change the DbgPrint mask so you see tracing information....ed nt!Kd_DEFAULT_Mask 0xffffffff), load cmdtrees, etc.

http://yeilho.blogspot.co.uk/2012/10/windbg-init-script.html
Take Control of WinDBG

An example script:

$$ Include a directory to search for extensions
$$ (point to a source controlled or UNC common directory so that all developers get access)
.extpath+"c:\svn\DevTools\WinDBG\Extensions"
$$ When debugging a driver written with the Windows Driver Framework/KMDF
$$ load this extension that comes from the WinDDK.
!load C:\WinDDK\7600.16385.1\bin\x86\wdfkd.dll
!wdftmffile C:\WinDDK\7600.16385.1\tools\tracing\i386\wdf01009.tmf
$$ load some extensions
.load msec.dll
.load byakugan.dll
.load odbgext.dll
.load sosex
.load psscor4
$$ Make commands that support DML (Debugger Markup Language) use it
.prefer_dml 1
.dml_start
$$ Show NTSTATUS codes in hex by default
.enable_long_status 1
$$ Set default extension
.setdll psscor4
$$ Show all loaded extensions
.chain /D
$$ Load some command trees
.cmdtree c:\svn\DevTools\WinDBG\cmdtree\cmdtree1.txt
.cmdtree c:\svn\DevTools\WinDBG\cmdtree\cmdtree2.txt
$$ Show some help for the extensions
!wdfkd.help
!psscor4.help
.help /D

Command Cheat Sheets

Crash Dump Analysis Poster v3.0
SOS Cheat Sheet (.NET 2.0/3.0/3.5)
WinDbg cheat sheet (Art of Dev)
WinDbg Kernel-Mode Extension Commands Flashcards

Extensions

"Extensions" allow you to extend the range of commands/features supported inside WinDBG.

bigLasagne (bldbgexts & blwdbgue)
- assembly syntax highlighting and a driver mapping tool)
BigLib Number Reader
Byakugan
- detect antidebugging methods, vista heap visualization/emulation, track buffers in memory
Call Flow Analyzer + KnExt
CmdHist
- records every command you executed in your debug session so you can re-execute easily
Core Analyzer
- check heap structures for corruption, detect objects shared by threads, etc
dom WinDBG Extension
- (!stlpvector, !idt, !unhex, !grep, etc)
dumppe
- dumps PE file from memory
Image Viewer Extension (Vladimir Vukićević)
Intel UEFI Development Kit Debugger Tool
- debug UEFI firmware
leaktrap
- GDI/USER handle tracker to aid in leak detection
Mona (requires PyKD)
- set of commands to aid in advanced analysis/find exploits
MSEC
- provides automated crash analysis and security risk assessment
narly
- lists info about loaded modules such as if using SafeSEH, ASLR, DEP, /GS (Buffer Security Checks)
netext (Rodney Viana)
- (!wservice - list WCF service objects, !wconfig - show .config lines, !whttp - list HttpContexts, !wselect/!wfrom - support SQL like queries on arrays)
ODbgExt
- open debugger extensions
OllyMigrate
- pass debuggee to another debugger without restarting
Psscor2
- a superset of SOS for assisting in debugging .NET 2.0 managed code
Psscor4
- a superset of SOS for assisting in debugging .NET 4 managed code
PyDBGExt
- allows python scripting to be used
PyKD
- allows Python to be used to script WinDBG
sdbgext (Nynaeve)
-(!valloc, !vallocrwx, !heapalloc, !heapfree, !remotecall, !remotecall64, !loaddll, !unloaddll, !close, !killthread, !adjpriv, !ret)
SieExtPub
-legacy extension...now built into WinDBG in ext.dll
SOSEX
- more commands for helping to debug managed NET 2.0 or 4.0 code
SPT/SDBGExt2 (Steve Niemitz)
- (!DumpHttpContext, !DumpASPNetRequests, !DumpSqlConnectionPools, !DumpThreadPool, etc)
Uniqstack
- source to a debugger extension (need an OSR Online account to access it)
viscope
- code coverage graph
Wait Chain Traversal/wct.dll (Codeplex Debugging Extensions
- display wait chains of application threads (helps find deadlocks)
windbgshark
- integrates Wireshark protocol analyser to enable VM traffic manipulation and analysis
WinDBG Extensions (Sasha Goldstein)
- Tracer, WCT, heap_stat, bkb, traverse_map, traverse_vector)
WinDBG Highlight (ColorWindbg.dll) [Use Google Translate to translate link]
- asm syntax highlighting

Write your own extension

Tools of the Trade: Part IV - Developing WinDbg Extension DLLs
The Basics of Debugger Extensions: Short Term Effort, Long Term Gain

Using WinDBG to Debug Managed Code

Breaking on an Exception
Breaking on specific CLR Exception
Debugging .Net framework source code within Windbg
Debugging exceptions in managed code using Windbg
Debugging managed code using WinDbg and SOS.dll
Debugging with WinDbg. Deadlocks in Applications.
MANAGED DEBUGGING with WINDBG. Introduction and Index
Setting .NET breakpoints in Windbg for applications that crash on startup

Scripting (C#, PS, Python, WinDBG)

KDAR (Kernel Debugger Anti Rootkit)
- a collection of WinDBG scripts
Sysnative BSOD Scripts/Processing Apps
WinDBG Script library
- a collection of WinDBG scripts
Scripting MDbg and DbgHostLib
- allows managed code to script the Managed Debugger (MDBG) and the DbgEng
ExtCS
- allows control of WinDBG via C# scripts
PowerDBG
- allows control of WinDBG via Powershell scripts
Pykd
- allows control of WinDBG via Python scripts
windbglib
- python wrapper library around the pykd extension for WinDBG, mimicking immlib (so you can use scripts originally written for Immunity Debugger)

Debuggers/Tools that use the dbgeng.dll API/WinDBG Tools

A Simple Dbgeng Based User Mode Debugger
Acorns.Debugging NET Deadlock Detector (uses cdb.exe) (download)
CLR Managed Debugger (MDBG)
DbgHost - How to control a debugging engine
Debug Diagnostic Tool v1.2 (DebugDiag), Ver 2.0 + DebugDiag Blog
Dynamorio - dynamic binary instrumentation tool which can interact with WinDBG
IDA + WinDBG plugin
GUI WinDBG
LeakShell (find managed leaks)
mdbglib - Managed Debug API
PyDbgEng
- python wrapper for Windows Debugging Engine
SOSNET - a WinDBG Fork/alternative shell that concentrates on using the SOS extension and supports C# scripting
SOSNET O2 fork - fork of SOSNET that uses Rosyln for the C# REPL (read-eval-print-loop) scripting engine
VDB/Vivisect (kenshoto) - provides a cross-platform debugging API layered on WinDBG
WinAppDbg + Heappie-WinAppDbg
Writing a basic Windows debugger

Different Ways to Generate Crash Dump Files for Post-Mortem Analysis

DebugDiag 2.0
Dump Cheat Sheet
- includes how to generate dump from Hyper-V, VMWare ESX, and XenServer VMs.
Citrix SystemDump
Keyboard Keypress Combination
MiniDumpWriteDump
- (via WIN32 API call inside your application). (Example for C# applications)
NMI Switch, or (here)
(hardware based feature to generate an NMI...usually found on high-end servers e.g. HP or you can obtain an add-in PCI card "Universal PCI Dump Switch"). Microsoft NMI technology background.
Procdump
System|Advanced System Settings|Startup and Recovery
(registry info),
(how to configure a Complete (Full) Memory Dump),
(how to enable Complete Memory Dump),
(how to enable Complete Memory Dump on Windows 7 when PC has lots of memory...normally not available when more than 2GB of memory)
Task Manager "Create Dump File"
UserDump, instructions (very old tool)
UserModeProcessDumper, instructions
Visual Studio "Save Dump As…"
WER (Windows Error Reporting....local dumps)
WinDBG

Dump Analysis Tools

BlueScreenView - finds the minidump .dmp files saved by Windows after a BSOD, and extracts information about what caused the crash
Debug.Analyzer (can analyse dump files and plug-ins can be written in .NET)
SAD - Simple After Dump (postmortem analyzer)
Volatility - framework for analyzing "memory" recorded in dump files (cheat sheet)

Dump related Tools

Citrix dumpcheck - checks consistency of dump file (looks like it's been abandoned link + link)
dumpchk (part of Debugging Tools) - checks consistency of a Dump file
MoonSols Windows Memory Toolkit (formerly windd) - converts various raw memory dump files into WinDBG compatible dmp files
vm2dmp - Microsoft Hyper-V VM State to Memory Dump Converter
vmss2core - converts VMWare snapshot file into a core dump file (download), (instructions)

Kernel Debugging Virtual Machines

VMKD - Virtual Machine KD Extensions
VirtualKD - (kernel debugger support for OS's hosted in VMWare/VirtualBox)

Videos

.NET Cracking 101 #2 - WinDbg basics
.NET Debugging for the Production Environment (Channel9)
dotnetConf - Advanced Debugging with WinDbg and SOS
David Truxall "Debugging with WinDBG"
Mike Taulty Debugging Memory Leaks
oredev 2009 Session: Debugging .NET Applications with WinDbg
Pluralsight Advanced Windows Debugging
(plus various other ones at Pluralsight)
Tess Ferrandez WinDBG (Channel9)

Blogs

Some blogs (mixture of native and managed code debugging).

Advanced .NET Debugging
All Your Base Are Belong To Us (Sasha Goldshtein)
Analyze-v
ASP.NET Debugging
Cyberiafreak (threading and advanced windows prog and debugging)
Debug Analyzer.NET
Debug and Beyond
Debugging Experts Magazine Online
Debugging Toolbox (Windbg scripts, debugging and troubleshooting tools and techniques to help you isolate software problems.)
Decrypt my World
greggm's WebLog
Junfeng Zhang's Windows Programming Notes
Kristoffer's tidbits
Mark Russinovich's Blog
Mike Stalls .NET Debugging Blog
Naveen's Blog
Never Doubt Thy Debugger (Carlo)
Notes from a Dark Corner
Ntdebugging Blog (Microsoft Global Escalation Services team)
Nynaeve. Adventures in Windows debugging and reverse engineering
PFE Developer Notes for the Field
Visual Studio Debugger Team
WinDbg by Volker von Einem

Advanced Articles and Tutorial Resources

Advanced Debugging Techniques in WinDbg
Debugging Applications for MS.Net and Windows (Powerpoint Slides)
Debugging STL Containers with WinDbg
Debug Tutorials 1-7 (CodeProject-Toby Opferman)
Debugging.tv
Developmentor WinDBG Tagged articles
Dr Fu's Security Blog - Malware Analysis Tutorials - Reverse Engineering Approach
Exploit writing tutorial part 5 : How debugger modules & plugins can speed up basic exploit development
Hunting Rootkits
Remote Microsoft Windows Server OS Kernel Debugging Using Dell Windows Debugger Utility (DWDU) (DELL(TM) Windows(R) Debugger Utility 1.1 README)

Alternative Debuggers

Bokken - (Inguma) (GUI for radare)
BugDbg
Debug++ (not released yet)
Debuggy
Discoloured Ring 0 Debugger (download)
edb (Linux)
FDBG
GoBug
Hades (Ring 3 debugger with anti debugger detection strategy)
Hopper (Linux, OSX and Windows) (Windows debugging not currently implemented)
Hyperdbg
IDA Debugger
ImmunityDebugger
Nanomite
Obsidian (non-intrusive debugger)
OllyDBG
PEBrowse
RaceVB6 (VB6 P-Code debugger)
radare
radare2ui (GUI for radare)
Rasta Ring 0 Debugger (RR0D)
Syser Kernel Debugger
TRW 2000 (very old debugger circa W9x) + dions plugin archive
VisualDux Debugger
Wintruder (extendable debugger)
WKTVDebugger (a debugger for Visual Basic P-Code) (download)
x64_dbg
Zeta Debugger