Debugging Windows processes
Last week I had to spend most of my time figuring out why an ASP .NET application CPU utilization hits 100%, I ended up learning how to take memory dumps and how to analyze them. Below is a summary of what I've learned.
Process of debugging using Win Debug/ADPlus( "C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe")
This is for debugging a 64 bit .NET process with .NET CLR version 2.0.
You can also use DebugDiag ("C:\Program Files\DebugDiag\DebugDiag.exe") to take memory dumps and analyzing them in a more user friendly manner.
1- Attach to process
- Attaching to process will cause freezing the process.
You can Use iisapp command line to resolve wpw3 processes and the app pools!threads
- Hitting “g” for F5 unfreezes the process
- Detach once you are done
- Stop will kill the process
2- Take a memory dump (freezes the process) , There are different types of dumps
1- Full dump : when process is failed and on 100% CPU
.dump /ma c:\dump_05192010_2PM.dmp
2- mini dumps: when you want to quickly attach to the process and get a quick memory dump and detach
adpPlus –quick –p process id –o outputPtath
adpPlus is installed with windebug and could be found in
“c:\program files(x86)\windebug” or "C:\Program Files\Debugging Tools for Windows (x64)\ " folder
3- Analyze the memory dump
1- Load up the memory dump
1. Open dump.
2. Set symbol path
To : srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
3. Set image path: this path points to a folder which holds referenced Dll’s , this is usually needed when analyzing a in a mini dump
2- Load SOS dll
1. .load C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll
2. .loadby sos mscorwks
3- Analyze by querying the memory dump
Things to look for in a memory dump analysis, all we have is call stacks and objects in memory (stack/heap).Stacks and memory objects could be viewed in two ways .NET CLR view of unmanaged view, most of below commands have an unmanaged version which outputs more detailed information regarding either call stack or objects.
1- Threads
a. List of Threads: command for this is !threads
b. to find out exactly what the CPU-usage was at the time the dump was taken !threadpool
c. This is a nice command that will list all running threads and their CPU-usage. It's your best friend when troubleshooting a high CPU hang issue. !runaway
2- Call stack(s):
you can view call stack of a particular thread or all call stacks of the process
To view call stack of a thread:
1- Go to the thread by command: ~[threaded]s : example : ~20s
2- Run: !clrstack or !clrstack -p
kb is its native version run ~* kb 2000 to get all native stacks
To view call stack of all threads:
~*e !clrstack (~*e kb is the unmanaged version)
3- Memory objects:
1- Content of heap memory : !dumpheap
2- View list of objects that are used by call stack: !dumpstackobjects
3- To view content/value of a object in heap or stack object run: !do [memory address]
4- Determine the ID of the thread owning the lock
!syncblk
Quick reference list of Windbg commands :
• .cls : clear screen
• ~*: Native threadlist
• ~* kb: Native call stack
• !clrstack –a:
• Analyze –v
• Symbol : srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
• .load C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll
• .loadby sos mscorwks
• !threads
Get on a thread: ~threadIds : eg. ~20s
• ~*e !clrstack
• !dumpheap
• !dumpheap -stat
• !dumpstackobjects == !dso
!syncblk
Quick reference list of Windbg commands :
• .cls : clear screen
• ~*: Native threadlist
• ~* kb: Native call stack
• !clrstack –a:
• Analyze –v
• Symbol : srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
• .load C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll
• .loadby sos mscorwks
• !threads
Get on a thread: ~threadIds : eg. ~20s
• ~*e !clrstack
• !dumpheap
• !dumpheap -stat
• !dumpstackobjects == !dso
Links:
