Operations

Debugging Windows applications – my walkthrough path

Recently faced an issue where a client had a memory allocation issue on one of their servers. I’m not going to deep dive into any of these, but here were some of the tools I used in tracking down the culprit:

  1. Go through Eventvwr and look at any error messages. There’s a list of error codes on MSDN.
  2. Get a process dump (full please!) using procdump. Configure with –ma –x to capture a dump on failure.
  3. In Windbg, open the crash dump and use !analyze –v. There’s an extensive set of help files on windbg.
  4. DebugDiag for crash analysis, slow performance, memory leak analysis, and performance analysis.
  5. ApplicationVerifier (appverif.exe) – for subtle programming errors. (i.e. heap corruption, incorrect handles). Since this must be run client-side, more of a dev than a production tool. This doesn’t require a process dump.
  6. Perhaps look at a self dump generation in code.
  7. Perfmon/PAL to look at memory leaks. This is for growing heaps (where memory is allocated but not deallocated), handle leaks (handles are created but not freed), and rising thread count. Memory allocation issues are very troublesome to catch and there’s a ton of third party tools out there to help – think RationalPurifier or Insure++.

 

In a little more detail:

  • For a crash:
    • Windbg debugger is xcopy deployable.
    • DebugDiag runs as a service, and monitors the process – if it crashes, it creates a dump.
    • Adplus – this is a command line tool you can set with –crash –p{processID}
  • For a hung process:
    • Task Manager (right mouse click to create a crash dump)
    • DebugDiag (process tab, rt-mouse click, create full user dump)
    • Adplus –hang –p{pid}
  • For a memory leak:
    • CLR memory profiler
    • DebugDiag (our Swiss army knife!) – leak track, rule and user dump
    • Umdh (old school – command line)
    • For .NET, there’s sos.dll and psscor4.dll – these are debugger extensions to analyze .NET dump stacks.
Advertisement

Perfmon and PAL notes

PAL is a tool (available on CodePlex –it’s open source) that generates an HTML report that charts performance counters generated from PerfMon and throws alerts when they’re exceeded. We’re not talking about anything obtrusive here – just a little VBScript GUI on top of Powershell to generate a nice-looking graphical UI. Its really sweet and can save you a ton of time in figuring out what the heck is going wrong when your server isn’t working properly. I ran it this morning on my underpowered little laptop and found a lot of issues with context switching for example:

Simple Overview:

  1. Download PAL and the separate Chart Controls for .NET 3.5 and install.
  2. Open up PAL and in the Threshold File tab export a template. Go ahead and view the XML file you generate in any text editor and view the counters. See, no magic!
  3. Open up Perfmon and create a new user defined data collector set. Import the file you created in #2. Run it in perfmon for at least 10-30 minutes.
  4. Go back to PAL and select the .blg file you just generated in #3 in the Counter Log tab. Click Next and then – last tab – generate your awesome HTML file, complete with charts and RYG indicators.

 

Long and Boring:

To run these tools, use the following steps:

  1. Download PAL from CodePlex and run the MSI package.
  2. Install the Chart Controls for .NET Framework 3.5 – there’s a link in the main codeplex page here.
  3. Now run PAL. Choose the default template:

     

    All done with that? Good.

     

    In the Threshold File page notice the long list of threshold files you can choose from – anything from BizTalk to System Overview (good if you have no specific match) to ASP.NET to SQL Server 2008R2 etc. BEFORE you do this go to the Questions tab and make sure its set properly for your OS/# of CPU’s / available memory etc. Then go back to Threshold and select Export to Perfmon Template File.

     

     

     

    And click on the Export to Perfmon template file and enter a nice-sounding name to create a perfmon profile template (BLG) onto your desktop. You’ll be using this next.

     

     

     

  4. Now open up Perfmon and expand the Data Collector Sets node, and select User Defined, and create a new data set.

Give it a name:

You’ll see the following – use the Browse button to select your template:

… and run it. Give it a healthy 10 minutes or even longer – you won’t stress your system. Then reenter PAL.

 

The tab above is worth a mention. Just select AUTO. You really don’t want to have it as more than that – a little too coarse of a grain to capture issues there – and anything less is WAY too fine. Trust me, with the number of counters we’re capturing, even with a very modest set it took PAL almost 12 minutes to finish compiling its report. 30 seconds is fine!

 

Everything else leave as is. I like to set the PAL tool so it Executes and Restarts:

 

There’s volumes of information here – a wealth of diagnostic information. Best part is, there’s KB links right there to point you to where to go. No more wondering what counters to pick, or what the values mean – it’s there and displayed, over time, in an easy to digest format. You can run perfmon just fine on any production environment with one caveat – BE CAREFUL about the length of time you’re running perfmon, usually 10-30 minutes is fine, although I’ve run it overnight – and don’t change it to something crazy like every second or something. Use the AUTO setting (which is 30 seconds). Changing this value to something unreasonable can bottom out your servers in no time flat, and makes root cause analysis harder.

There’s another tool out there called Server Performance Advisor – it analyzes both perfmon logs and Event Tracing for Windows. It’s best for analyzing short term performance problems; PAL is best for covering long periods of time.

 

Helpful links: