Debugging Windows processes

Debugging Windows processes

Last week I had to spend most of my time figuring out why an ASP .NET application CPU utilization hits 100%, I ended up learning how to take memory dumps and how to analyze them. Below is a summary of what I've learned.

Process of debugging using Win Debug/ADPlus( "C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe")

This is for debugging a 64 bit .NET process with .NET CLR version 2.0.

You can also use DebugDiag ("C:\Program Files\DebugDiag\DebugDiag.exe") to take memory dumps and analyzing them in a more user friendly manner.

1-      Attach  to process

-          Attaching to process will cause freezing the process.

You can Use iisapp command line to resolve wpw3 processes and the app pools!threads

-          Hitting “g” for F5 unfreezes the process

-          Detach once you are done

-          Stop will kill the process

2-      Take a memory dump (freezes the process) , There are different types of dumps

1-      Full dump : when process is failed and on 100% CPU

.dump /ma c:\dump_05192010_2PM.dmp

2-      mini dumps: when you want to quickly attach to the process and get a quick memory dump and detach

adpPlus –quick –p process id –o outputPtath

adpPlus is installed with windebug and could be found in

“c:\program files(x86)\windebug” or  "C:\Program Files\Debugging Tools for Windows (x64)\ " folder

3-      Analyze the memory dump

1-      Load up the memory dump

1.       Open dump.

2.       Set symbol path

To : srv*c:\websymbols*http://msdl.microsoft.com/download/symbols

3.       Set image path: this path points to a folder which holds referenced Dll’s , this is usually needed when analyzing a in a mini dump

2-      Load SOS dll

1.       .load C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll

2.       .loadby sos mscorwks

3-      Analyze by querying the memory dump

Things to look for in a memory dump analysis, all we have is call stacks and objects in memory (stack/heap).Stacks and memory objects could be viewed in two ways .NET CLR view of unmanaged view, most of below commands have an unmanaged version which outputs more detailed information regarding either call stack or objects.

1-      Threads

a.       List of Threads: command for this is !threads

b.      to find out exactly what the CPU-usage was at the time the dump was taken !threadpool

c.       This is a nice command that will list all running threads and their CPU-usage. It's your best friend when troubleshooting a high CPU hang issue.  !runaway

2-      Call stack(s):

 you can view call stack of a particular thread or all call stacks of the process

To view call stack of a thread:

1-      Go to the thread by command: ~[threaded]s : example : ~20s

2-      Run: !clrstack or !clrstack -p

kb is its native  version run  ~* kb 2000 to get all native stacks

                                       To view call stack of all threads:

~*e !clrstack (~*e kb is the unmanaged version)

3-      Memory objects:

1-      Content of heap memory :  !dumpheap

2-      View list of objects that are used by call stack: !dumpstackobjects

3-      To view content/value of a object in heap or stack object run: !do [memory address]

4-      Determine the ID of the thread owning the lock
!syncblk

Quick reference list of Windbg commands :
•    .cls : clear screen
•    ~*:  Native threadlist
•    ~* kb: Native call stack
•    !clrstack –a:
•    Analyze –v
•    Symbol : srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
•    .load C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll
•    .loadby sos mscorwks
•    !threads
    Get on a thread: ~threadIds : eg. ~20s
•    ~*e !clrstack
•    !dumpheap
•    !dumpheap -stat
•    !dumpstackobjects == !dso

Links:

http://blogs.msdn.com/johan/archive/2007/11/13/getting-started-with-windbg-part-i.aspx

http://blogs.msdn.com/tess/pages/net-debugging-demos-information-and-setup-instructions.aspx

http://blogs.msdn.com/tess/archive/2008/02/06/net-debugging-demos-lab-1-hang-review.aspx

http://blogs.msdn.com/tess/archive/2008/02/22/net-debugging-demos-lab-4-high-cpu-hang.aspx

Exceptions

Exception Throwing

Exception throwing guidelines described in this section require a good definition of the meaning of execution failure. Execution failure occurs whenever a member cannot do what it was designed to do (what the member name implies). For example, if OpenFile method cannot return an opened file handle to the caller, it would be considered an execution failure.

Most developers have become comfortable with using exceptions for hard error cases such as division by zero or null references. In the Framework, exceptions are used for both hard errors and logical errors. At first, it can be difficult to embrace exception handling as the means of reporting all functional failures. However, it is important to design all public methods of a framework to report method-failures by throwing an exception.

There are a variety of excuses for not using exceptions, but most boil down to the two perceptions that exception handling syntax is undesirable, so returning an error code is somehow preferable, or that a thrown exception does not perform as well as returning an error code. The performance concerns are addressed in the performance section below. The concern over syntax is largely a matter of familiarity and should not be a consideration. As an API designer we should not make assumptions about the familiarity of the application developers consuming our code. 

Do not return error codes. Exceptions are the primary means of reporting errors in frameworks.

Do report execution failures by throwing exceptions. If a member cannot successfully do what is designed to do, it should be considered an execution failure and an exception should be thrown.

Consider terminating the process by calling System.Environment.FailFast (.NET Framework 2.0 feature) instead of throwing an exception, if your code encounters a situation where it is unsafe for further execution.

Do not use exceptions for normal flow of control. Except for system failures, there should generally be a way to write code that avoids exceptions being thrown. For example, you can provide a way to check preconditions before calling a member to allow users to write code that does not throw exceptions.

ICollection collection = …

if(!collection.IsReadOnly){

    collection.Add(additionalNumber);

}

The member used to check preconditions of another member is often referred to as a tester and the member that actually does the work is called a doer. See performance section below for more information on the Tester-Doer Pattern.

There are cases when the Tester-Doer pattern may have an unacceptable performance overhead. In such cases the so called TryParse Pattern (see section below) should be used.

Consider performance implications of throwing exceptions. See section below for details.

Do document all exceptions thrown by publicly callable members because of a violation of the member contract (rather than a system failure) and treat them as part of your contract. Exceptions that are a part of the contract should not change from one version to the next.

Do not have public members that can either throw or not based on some option.

Type GetType(string name, bool throwOnError)

Do not have public members that return exceptions as the return value or an out parameter.

Do set all the relevant properties of the exception you throw.

Consider using exception builder methods. It is common to throw the same exception from different places. To avoid code bloat, use helper methods that create exceptions and initialize their properties. For example:

class File{

   string fileName;

   public byte[] Read(int bytes){

      if (!ReadFile(handle, bytes))

            throw NewFileIOException(...);

   }

   FileException NewFileException(...){

      string description = // build localized string

      return new FileException(description);

   }

}

Do not throw exceptions from exception filter blocks. When an exception filter raises an exception, the exception is caught by the CLR, and the filter returns false. This behavior is indistinguishable from the filter executing and returning false explicitly and is therefore very difficult to debug.

Avoid explicitly throwing exceptions from finally blocks. Implicitly thrown exceptions resulting from calling methods that throw are acceptable. 

Link to above Blog

Designing Custom Exceptions

In some cases, it will not be possible to use existing exceptions. In those cases, you’ll need to define custom exceptions. The guidelines in this section provide help on doing that.

Avoid deep exception hierarchies. 

Do derive exceptions from System.Exception or one of the other common base Exceptions.

Do end exception class names with the ‘Exception’ suffix.

Do make exceptions serializable. An exception must be serializable to work correctly across application domain and remoting boundaries.

Do provide (at least) these common constructors on all exceptions. Make sure the names and types of the parameters are exactly as in the example below.

public class SomeException: Exception, ISerializable {

   public SomeException();

  public SomeException(string message);

  public SomeException(string message, Exception inner);

  // this constructor is needed for serialization.

   protected SomeException(SerializationInfo info, StreamingContext context);

}

Do report security sensitive information through an override of ToString only after demanding an appropriate permission.

If the permission demand fails, return a string excluding the security sensitive information.

Annotation (Rico Mariani):

Do not store the results of ToString in any generally accessible data structure unless that data structure suitably secures the string from untrusted code.  This advice applies to all strings but since exception strings frequently contain sensitive information (such a file paths) I reiterate the advice here.

Do store useful security sensitive information in private exception state. Ensure only trusted code can get the information.

Consider providing exception properties for programmatic access to extra information (besides the message string) relevant to the exception.

Application Exception :

http://blogs.msdn.com/kcwalina/archive/2006/06/23/644822.aspx

http://msdn.microsoft.com/en-us/library/seyhszts.aspx

http://msdn.microsoft.com/en-us/library/5b2yeyab(v=VS.90).aspx

C# Generic Dictionary retrieval performance benchmarks

C# Dictionary select performance benchmarks

These benchmarks are testing a Dictionary <int,string> and a  Dictionary<IUserKey, string>

In theory retrieval on a Hash Table (which is the data container inside each dictionary) should be constant regardless of number of items in the Hash Table.

The reason being is that accessing a hash table is a random access of an array element  which of the key is generated by a Hashing algorithm.

1-      Dictionary.GetItem(key)

2-      f(Key) à indexValue: random access index of value. F is the hash algorithm.

3-      Dictionary.hashtable[indexValue]

However, The problem is that the hash algorithm is not generating a unique index all the time, when it doesn’t it a collision event!

Collisions are the culprit of decrease in hash table item retrievals (long story short)

In these set of tests I am examining the different items added to dictionary an int an object and a Object with specified HashCode algorithm.

DictionaryPerfTest10Mil1Mil means that Test was ran over a dictionary filled with 10 million items and it was queried 1 million times

Int dictionary-  

  private static void IntDictPerf(int fillSize, int selectionSize)

        {

            Random random = new Random();

            Dictionary<int, string> dictHash = new Dictionary<int, string>(fillSize);

            try

            {

                for (int i = 0; i < fillSize; i++)

                {

                    dictHash.Add(i, "value" + i.ToString());

                }

            }

            catch (Exception )

            {

                System.Diagnostics.Debug.WriteLine("Collision occured");

            }

            Stopwatch stopWatch = new Stopwatch();

            stopWatch.Start();

            for (int i = 0; i < selectionSize; i++)

            {

                int index = random.Next(fillSize);

                var x = dictHash[index];

            }

            stopWatch.Stop();

            TimeSpan ts = stopWatch.Elapsed;

            string elapsedTime = String.Format("{0:00}:{1:00}:{2:00}.{3:00}",

                ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10);

            System.Diagnostics.Debug.WriteLine(elapsedTime, "RunTime");

        }

Result:

Under 200K items fetch time is almost constant above 300K to 500K it doubles and above 500K it triples.

1^st Run:

IntDictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.27

IntDictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.20

IntDictionaryPerfTest500K1Mil - RunTime: 00:00:00.15

IntDictionaryPerfTest300K1Mil - RunTime: 00:00:00.11

IntDictionaryPerfTest200K1Mil - RunTime: 00:00:00.06

IntDictionaryPerfTest100K1Mil - RunTime: 00:00:00.06

IntDictionaryPerfTest10K1Mil - RunTime: 00:00:00.05

IntDictionaryPerfTest1K1Mil - RunTime: 00:00:00.04

2^nd Run:

IntDictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.27

IntDictionaryPerfTest500K1Mil - RunTime: 00:00:00.15

IntDictionaryPerfTest300K1Mil - RunTime: 00:00:00.09

IntDictionaryPerfTest100K1Mil - RunTime: 00:00:00.05

IntDictionaryPerfTest10K1Mil - RunTime: 00:00:00.05

IntDictionaryPerfTest1K1Mil - RunTime: 00:00:00.04

3^rd Run:

IntDictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.27

IntDictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.21

IntDictionaryPerfTest500K1Mil - RunTime: 00:00:00.16

IntDictionaryPerfTest100K1Mil - RunTime: 00:00:00.05

IntDictionaryPerfTest10K1Mil - RunTime: 00:00:00.05

4^th Run:

DictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.27

DictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.20

DictionaryPerfTest100K1Mil - RunTime: 00:00:00.05

DictionaryPerfTest10K1Mil - RunTime: 00:00:00.05

5^th Run:

DictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.27

DictionaryPerfTest100K1Mil - RunTime: 00:00:00.05

DictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.21

DictionaryPerfTest10K1Mil - RunTime: 00:00:00.05

6^th Run:

DictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.27

DictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.21

DictionaryPerfTest100K1Mil - RunTime: 00:00:00.05

DictionaryPerfTest10K1Mil - RunTime: 00:00:00.05

DictionaryPerfTest1K1Mil - RunTime: 00:00:00.04

User Key struct - Default hashcode (GetHashCode())

public interface IUserKey

    {

        int ID { get; set; }

    }

    public struct UserKey : IUserKey

    {

        public int ID

        {

            get;

            set;

        }

       // public override int GetHashCode()

        //{

        //    return this.ID.GetHashCode();

        //}

    }

private static void UserKeyDictPerf(int fillSize, int selectionSize)

        {

            Random random = new Random();

            Dictionary<IUserKey, string> dictHash = new Dictionary<IUserKey, string>(fillSize);

            try

            {

                for (int i = 0; i < fillSize; i++)

                {

                    IUserKey usr = new UserKey() { ID=i };

                    dictHash.Add(usr, "value" + i.ToString());

                }

            }

            catch (Exception )

            {

                System.Diagnostics.Debug.WriteLine("Collision occured");

            }

            Stopwatch stopWatch = new Stopwatch();

            stopWatch.Start();

            for (int i = 0; i < selectionSize; i++)

            {

                //int index = random.Next(fillSize);

                IUserKey usr = new UserKey() { ID = random.Next(fillSize) };

                var x = dictHash[usr];

            }

            stopWatch.Stop();

            TimeSpan ts = stopWatch.Elapsed;

            string elapsedTime = String.Format("{0:00}:{1:00}:{2:00}.{3:00}",

                ts.Hours, ts.Minutes, ts.Seconds, ts.Milliseconds / 10);

            System.Diagnostics.Debug.WriteLine(elapsedTime, "RunTime");

        }

DictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.80

DictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.47

DictionaryPerfTest100K1Mil - RunTime: 00:00:00.29

DictionaryPerfTest10K1Mil - RunTime: 00:00:00.16

DictionaryPerfTest1K1Mil - RunTime: 00:00:00.14

User Key struct - Overriden hash code

Same as the above test with the exception  of defining the HashCode method

    public struct UserKey : IUserKey

    {

        public int ID

        {

            get;

            set;

        }

        public override int GetHashCode()

        {

            return this.ID.GetHashCode();

        }

    }

DictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.78

DictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.48

DictionaryPerfTest100K1Mil - RunTime: 00:00:00.30

DictionaryPerfTest10K1Mil - RunTime: 00:00:00.17

DictionaryPerfTest1K1Mil - RunTime: 00:00:00.15

User Lo Key struct –

UserLoKeyDictionaryPerfTest10Mil1Mil - RunTime: 00:00:00.79

UserLoKeyDictionaryPerfTest1Mil1Mil - RunTime: 00:00:00.48

UserLoKeyDictionaryPerfTest100K1Mil - RunTime: 00:00:00.28

UserLoKeyDictionaryPerfTest10K1Mil - RunTime: 00:00:00.16

UserLoKeyDictionaryPerfTest1K1Mil - RunTime: 00:00:00.15