Application Insights

Monitoring, and Why It Matters To You.

I’ve done a few articles on Application Insights – (older ones here and here) – but none yet on Operations Management Suite, because 1) I’m not IT in my background and 2) I’m busy leave me the heck alone! (I kid, we’ll get to it eventually.) These have all been more how-to – and admittedly it’s so easy I hesitate to call it that – versus why to. But last night on a plane coming back from Kansas City, I was mulling this over. (It helps that I had the excellent if somewhat clunky “The Art of Monitoring” on my Kindle.)

Monitoring has long been the secret sauce of DevOps. How else do we get feedback on our priorities, and actual metrics – not guesses – on which features are in use? What’s often overlooked though is that it can actually help you fight back against the wrong kind of change management – one that increases your bureaucratic workload and actually makes your build riskier and harder to fix. How is that possible?

The Blame Game

Let’s start with some basic negative cycles we’ve all seen when there’s very visible production outages. When bad things happen in production, we immediately start seeing the oddest thing happen – the SDLC process starts to dissolve into this negative cycle of blame and recriminations.

Take the example of Knight Capital in 2012. My good friend Donovan Brown often cites this as a warning example. Here, one messy 15 minute deployment led to 440M loss. In the wake of a disaster like this, John Allspaw noted that there are two counterfactual narratives that spring up:

  1. Blame change control. “Hey, better CM practices could have prevented this!”
  2. Blame testing – “If we had better QA, we at least could have taken steps to detect it faster and recover!”

It’s hard to argue with either of these. And it’s true, the RIGHT kind of change controls do need to be implemented. But by clenching like this, as Gene Kim has noted in The DevOps Handbook, “in environments with low-trust, command and control cultures, the outcomes of their change control and testing countermeasures end up hurting more than they help. Builds become bigger, less frequent and more risky.” Why is this?

This is because the devs/QA team begins implementing increasingly more clunky testing suites that take longer to execute, or writing unit tests that frequently don’t catch errors in the user experience. In a pinch, the QA team begins adding a significant amount of manual smoketesting versus automated tests. Management begins imposing long and mandatory change control boards every week to approve releases and go over introduced defects from the previous week(s) – I’ve seen these groups grow into the 100’s, most of whom are very far removed from the application. More controls, remote gatekeepers and a manual approval process leads to increased batch sizes and deployment lead times – which reduces our chances of a successful deployment for both dev and Ops. Our feedback loop – the times stretch out, reducing its value. A key finding of several studies is that high performing orgs relied more on peer review and less on external approval of changes. The more orgs rely on change approval, the worse their IT performance in both stability (MTTR and change fail rate) and throughput (deployment lead times and frequency).

This is often where I tell the story of my dad and I, trying to cut down a few trees for my uncle that had fallen across a local creek in NW Washington in a storm. The river had risen several feet and we city boys were standing below the dam formed by these large tree trunks. I remember looking up at the water swelling and pushing against the trees, as we were cutting into them, and thinking what those several feet of water would do once released. It didn’t take a lot of imagination to picture the outcome – two idiots being swept out to the Pacific Ocean – but the problem was my uncle was standing a few dozen feet away, hand on his hips, watching us with his lips tight in a disapproving line. I told my father, “Dad, I don’t care what it takes, but we need to find a way of breaking that chainsaw!” That’s the kind of backlog that can form that can choke your release cycle, reducing flow and increasing build sizes and risk. (And, by accidentally dunking the chainsaw, we were able to successfully kill the project and earn the lasting contempt of my uncle – “I want to thank you boys – it’s been a long time since I’ve been to the circus!”)

Telemetry To The Rescue

The main issue above is that this overreactive organization was trying to prevent errors and bugs from happening. Sometimes, they even call their recap (punitive!) meetings “Zero Defect Meetings” – as if such a kind of operational perfection is attainable! In contrast, DevOps savvy companies don’t try to focus on MTBF – reducing their failure count. They know outages are going to happen. Instead, they try to treat each failure as an opportunity – what test was missing that could have caught this, what gap in our processes can address this next time? Especially they focus on improving their REACTION time – improving their time to recovery, MTTR (Mean Time to Recover). Testing and automated instrumentation – that famous passage about wanting “cattle not pets”, i.e. blowing away and recreating environments at whim – forms the heart of their adaptive, flexible response strategy.

Puppet Labs – in their excellent 2014 “State of DevOps” report – mentioned that organizations that want to improve on their reaction time (MTTR) benefit the most – and it’s not even close, by an order of magnitude – from two technical tools/approaches:

  • Use of version control for all production artifacts – When an error is identified in production, you can quickly either redeploy the last good state or fix the problem and roll forward, reducing the time to recover.
  • Monitoring system and application healthLogging and monitoring systems make it easy to detect failures and identify the events that contributed to them. Proactive monitoring of system health based on threshold and rate-of-change warnings enables us to preemptively detect and mitigate problems.

We’re going to talk about monitoring above. How can monitoring help turn the tide for us so we don’t overreact because of a production outage?

So above we can see a few fixes that can transform that reactive, vicious cycle into a responsive but measured virtuous cycle that addresses the core problems you’re seeing in PROD. Some are nontechnical or more process related than anything else – and note that fixing the issue starts with purity of code – as early in the process as possible:

  1. Adding or strengthening production telemetry (we can confirm if a fix works – and autodetect next time)
  2. Devs begin pushing code to prod (I can quickly see what’s broken and make decisions to rollback vs patch). Note on this, a rollback – going to a previous version – is almost always easier and less risky. But sometimes fixing forward and rolling out a change using your deployment process is the best way forward.)
  3. Peer reviews. This includes not just code deployments but ops/IT changes to environments! (remember the Phoenix project, 80% of our issues caused by unauthorized changes, often by IT to environments, 80% of our time stuck figuring out what in this soup of changes caused the issue – before we even lift a finger to resolve anything! I’ll write more about how to do a productive peer review – expecially pair programming, which is really a code review on programming – later.)
  4. Better automated testing (again, more on this later. Look at Jez Humble’s excellent Continuous Delivery or Agile Testing for more on this.
  5. Batch sizes get smaller. The secret to smooth and continuous flow is making small, frequent changes.

A key driver here though is information radiators- a term that actually comes from Toyota’s Lean principles. This creates a feedback loop, which broadcasts back issues as quickly as possible, radiating information out on how things are going.

Etsy – just to take one company as an example – takes monitoring so seriously that some of their architects have been quoted as saying their monitoring systems need to be more available and scalable than the systems they’re monitoring. One of their engineers was quoted as saying, “If Engineering at Etsy has a religion, it’s the Church of Graphs. If it moves, we track it. Sometimes we’ll draw a graph of something that isn’t moving yet, just in case it decides to make a run for it. Tracking everything is the key to moving fast, but the only way to do it is to make tracking anything easy. We enable engineers to track what they need to track, at the drop of a hat, without requiring time-sucking configuration changes or complicated processes.”

Another great thinker in the DevOps space, Ernest Mueller, has said – “One of the first actions I take when starting in an organization is to use information radiators to communicate issues and detail the changes we are making. This is usually extremely well received by our business units, who were often left in the dark before. And for Deployment and Operations groups who must work together to deliver a service to others, we need that constant communication, information and feedback.

I know I found that being true in my career. I discovered this fairly early on in my adoption of Agile with some sportswear companies here in the Oregon region. I worked for some very personality-driven orgs with highly charged, negative dynamics between teams. As I adopted Agile, which meant broadcasting honest retrospectives – including my screw-ups and failure to meet sprint goals – I expected a Donkey Kong type response and falling hammers. The most shocking thing happened though – the more brutally honest and upfront I was on what had gone wrong, I found myself having a better relationship with the business and my IT partners. And, mistakes we made on the team were owned up to – and they typically didn’t repeat, not without the group holding the culprit (including me) responsible. That kind of “government in the sunshine” type transparency and candor was the biggest single turning point of our Agile transformation.

It’s been said, rightly, that every lie we tell ourselves comes with a payoff and a price.
I believe that very much to be the case. For developers or IT, we’ve been very used to thinking we are AWESOME and WONDERFUL and the OTHER GUYS are cowboys/bureaucratic tools and are EVIL. Maybe that story – which has the short term payoff of making us feel virtuous – comes with a heavy price, of limiting our success in rolling out easy to manage and maintain applications and delivering business value faster. By using instrumentation and telemetry, we demonstrate that we are not lying to ourselves or to our customers/the business. And suddenly a lot of those highly charged, politically sensitive meetings you find yourself in lose a lot of their subjectivity and poison – the focus is on improving numbers versus the negative punish/blame scenario.

In Closing

  • Like testing, instrumentation and monitoring seems to be a bolt on or an afterthought in every project. That’s a huge mistake. Make instrumentation and metrics the backbone of your DevOps movement, as it’s the only thing that will tell you if you’re making specific progress and earn you credibility in the eyes of the business.
  • Don’t let your developers tell you that it’s too hard or have it be an afterthought. It takes just a few minutes to make your release and application availability metrics available to all.
  • And if your telemetry system is difficult to implement or doesn’t collect the metrics you need, think about switching. Remember the Etsy lesson – making it easy and quick is the way to go. (which is why I really like App Insights!)



Application Insights, and what it can do for you…

I’m giving some presentations over the next few weeks on dashboarding – specifically using Application Insights. I thought I would write up some of the things I discovered in doing some prep research, including a full walkthrough so you can try it on your own projects.

First off, I think most of us have fooled around with Application Insights – it’s a checkbox on creating a project in Visual Studio for gosh sake. And maybe, we got frustrated with some of its limitations – “It doesn’t aggregate well! It’s not customizable!” – and gave up in frustration. Well, we may have quit on it too soon. Microsoft is quite committed to it as a tool – and from what I’ve seen, compared to its very inflexible and stale early iterations, is light years ahead of what I thought. It’s easy, painless. In short, there is no longer any excuse for not dashboarding your code.

I’ll do a followup post in a few days on WHY this is important. For now, here’s some quick steps to try playing with it yourself.

Getting Started

You may want to begin with some videos to kickstart some interest. First, here’s some good videos to set the stage – here’s a good intro video, another on app availability, and another with a little more detail on usage monitoring.

But let’s step out of the documentation for a second and let’s talk demo.


Short List of Steps

  1. Build a MVC app in Visual Studio and publish it to Azure. Give it a very specific name and publish to East US. (at least, that’s what my subscription allowed!)
  2. Enable Application Insights.
  3. Open up Application Insights.config and look at the info. This is how you do addl perf counters. (see this API doc for more)
  4. R mouse click on the project and select Application Insights. This opens up the portal. Pin it to the dashboard.
  5. Click on some items here and explore modifying the chart, etc. Look at page view load times for example.
  6. Now, notice we don’t have usage data. Click on dashboard, click on Usage. Add this Js script to header on layout.cshtml. Now, when we rebuild, we’ve got actual usage times and can track it.
  7. If you wanted to do A/B testing, look at this page. Per this page, you can add tags to help segment out your errors. This can be done using either Javascript or C#/VB. This is also how you do A/B testing BTW. You put version numbers in the C#/Js. Overall in web.config
  8. Dashboard – cover:
    1. Metrics Explorer – drill down to server log telemetry
    2. Modifying charts (add Server Performance Counters)
    3. Add alerts (browser page load time)
    4. Application Map
    5. Availability (here is where you add a basic ping test every 5 minutes. Make sure you turn this off post-event!)
    6. Overview Timeline


In short, we’ll be covering MOST of the items below:

In Greater Length

Build a MVC app in Visual Studio and publish to Azure. Give it a funky name and publish to East US.

Enable Application Insights. If you have an existing app, no biggie – right mouse click on the project in Solution Explorer and enable it, then copy that single line of Javascript onto your page(s) as needed. In our case though, we’re checking the box to create the project with A.I. installed. That makes it soooo easy. And as lazy programmers – that’s what we want right?

You’ll notice there’s a very lightweight touchpoint here in your project – a few references and a new config file. Open up application insights.config and look at info:

Wow, that’s pretty easy! Check out the API I mentioned above in the short version for a list of all the goodies you can instrument/measure.

Now let’s check out our dashboard. R-mouse click on project – open up Application insights. This brings you to the Azure Portal. Notice you now have a new App Insights dashboard available to you. Go ahead and if you wish r-click to pin it to the dashboard.


Clicking on this reveals some interesting, built in instrumentation. I mean, look at all these goodies!

You can edit the chart – just click on the top right – to add some metrics that you care about.

Go to main dashboard. Notice we have no user data collection. This means we’re blind when it comes to geography/OS/browser – defining our users and the features they are liking. No biggie – clicking on the text in the chart where it says “click here to view more about usage data” or something like that – you’ll see a snippet of code available. Copy that – and let’s paste it into the _Layout.cshtml file header of our app, like so:

Per this page, you can add tags to help segment out your errors/operations. This can be done using either Javascript or C#/VB. This is also how you do A/B testing BTW. It’s extremely powerful – putting a parameter like “Version 2.1” for example. Now, you can tell when your newest version of the application is performing slower than it should for a subset of users on your production boxes. And, using deployment slots or feature toggling, you can safely kill it or fix in place without a widespread service outage.

Experiment with adding a web test as well – which is really where the rubber hits the road. Now you can have different geos- Asia, N America, etc – reporting on the true end user experience. No more guesswork about how your new app is doing in Japan for example!

Note here that you may have to remove spending limits (as I had to) with web tests.

On logging – this is a simple mod to a web.config. See this page for more on this – If you use NLog, log4Net or System.Diagnostics.Trace for diagnostic tracing in your ASP.NET application, you can send your logs to Application Insights – which can then be merged with the other telemetry coming from your application, so that you can identify the traces associated with servicing each user request, and correlate them with other events and exception reports.

I was very impressed with how quickly Application Insights could allow me to drill down to a particular event for example. Check this screenshot out below:


Click on Metrics Explorer. Experiment with drilling through to stack trace of event logs, adding tags.

Notice how sweet and easy it is to add alerts. Now I can finetune my alerts so they’re actually sending me valuable information – instead of it being “your dog has died” kind of dead data.

Finish editing the charts so that they are showing metrics on Usage like

  • Usage – Browser Page Load
  • Process IO Rate
  • Users


And I do love the simple, easy instrumentation that comes with web tests. Notice – this will cost you over time, so be sure to turn it off (disable) it if not in active use!


A little more detail on my web test settings, see below:

More For The Future

Application Insights redux

I spent a few minutes today getting reacquainted with Application Insights. Here’s some links and walkthrus so you can have fun with it yourself. For web dashboarding, I do think it has distinct advantages over Google Analytics etc.

See this article, which – fair warning – I borrowed from pretty heavily in my experiments. You’ll need an Azure account and at least VS2013 update 3.

There’s two ways of going about sprinkling ApplicationINsights pixie dust all over your app. (And no it doesn’t have to be ASP.NET site!)

Existing Site:

  1. Create a new project (or open an existing one).
  2. R-click on project and select Add Application Insights Telemetry. This will add all the references you’ll need and add some event handlers to your startup code.

Can add App Insights Status Monitor to an existing app (NuGet?) Executes NuGet, updates web.config, and restarts.

The other fancy way is on creation. Note, adding it to your site is now the DEFAULT! Redmond won’t rest till all your dashboarding/alerts are showing up in one central place.

From your Azure Portal:

Click New, Application Insights. You can browse to your site from here as well.


Once you’re viewing the A.I. portal, try clicking on Requests or Alerts to improve your telemetry/charting.

It was embarrassing how easy it was to add availability/performance tests using this site’s walkthrough. In my case, I had the site instrumented: all I had to do was click on the webtests tile in the AppInsights dashboard to create my test:


And wait a few minutes and presto – I had to manually refresh the portal – but I get fancy-dancy availability graphics. It’s drillable and … wow to get this much out of the box is kinda amazing.


Another option is to explore adding telemetry. Click on the Diagnostic Search icon in the portal…

Which opens up a drillable, filterable list of all the telemetry on my app. Including tags.


Go ahead and click on one of those events – I just dare ya. Notice you can drill in specifically by clicking on the three ellipses below the Request Details node to view every field collected:



In this case, using the very cool Filter set to Errors showed me exactly where the issue was – database availability. It even parses log4Net or Nlog traces.Easy peasy!

I could have added custom events using JavaScript, or maybe the following C# snippet:

// Set up some properties:

var properties =

{{“game”, currentGame.Name},
{“difficulty”, currentGame.Difficulty}};

var measurements =

{{“Score”, currentGame.Score},
{“Opponents”, currentGame.OpponentCount}};


// Send the event:

telemetry.TrackEvent(“endOfGame”, properties, measurements);


Alternately if I wanted to throw errors – this is very powerful as you can navigate between failed requests and exceptions and read the entire exception stack – create a new page called ThrowError or the like and add the following to Page_Load – after adding a using reference to Microsoft.ApplicationInsights in the header:

var telemetry = new



//doing some stuff here




catch (Exception ex)


// Set up some properties:

var properties = new
Dictionary <string, string>

{{“appinsightsdemo”, “usernamehere”}};


var measurements = new
Dictionary <string, double>

{{“Users”, 1}};


// Send the exception telemetry:

telemetry.TrackException(ex, properties, measurements);


Then I build it out and try hitting that page a few times. Suddenly I get drillable exceptions:

A wonderful overall video is shown here –


Getting started with app dashboarding using Application Insights.

Application Insights have been getting some buzz recently with TechEd and other events. Why is that?

Let’s start with an existing website. Once I installed the VISX file for Application Insights and restarted Visual Studio, I can right mouse click on my project and add Application Insights Telemetry to the project.


It looks like you MUST have your project hosted on Visual Studio Online for this to work. I dusted off my old account and…

Hmmm. What ARE these cool new dashboards that just appeared in my web project?

Let’s build it and deploy out to a website. We’re going to follow the step by step instructions here. (and… screeching halt – this won’t work for my app, since it’s an intranet app and not public-facing. Sigh.) Still, I’m going to leave this here since it shows – adding dashboarding to show availability and performance for your website is literally just a right-mouse click. You can check for:


track usage

…including what pages/features are being used by your customers:




.. and even set up tiles with a custom dashboard. Think about deploying this along with your site. I was so impressed with the real-world metrics that came OOTB with Azure. Microsoft’s definitely upped the ante with Application Insights.

Other Tidbits


Some good advice from the redoubtable Julie Lerman this month in MSDN. Updating from EF4 to EF6 (and now EF7 I guess!) is relatively trivial – mainly some namespace changes. But breaking up a large data model into pieces, while HIGHLY desirable from a performance standpoint, is a little more substantial – and changing from ObjectContext to the newer DbContext API is fairly involved. She recommends doing it one piece at a time. Start with a separate model project, split out as you like – then change your app references one by one and see what functionality breaks. There’s some big performance gains you’ll see in going from EF4 to 5/6; it’s well worth it. Please check out her books and teaching videos on Pluralsight, they’re terrific.

Side note – I thought this Grantland article was terrific. Nintendo started with a small market, kept a laser focus on a specific niche (9-to 14-year olds) and grew with one dynamic product, Donkey Kong. They survived and counterpunched Universal when they tried to shake them down with a copyright claim extortion from the 1933 movie King Kong. And they deliberately created marketing buzz with a specific look and only fulfilling a fraction of vendor requests. This long term strategy over short term profits – combined with the Seal of Quality and a strict licensing program that weeded out all but serious development companies – fueled their rise from the wreckage of the American video game companies like Atari.