Release Management

Starting and Scaling DevOps in the Enterprise – review

As many of you know, I’m a huge fan of the work Gary Gruver has done – in particular his book “Leading the Transformation” on his experiences at HP trying to transform a very traditional enterprise. (See my earlier mention of his book on this blog, here.) His newest work is out – Starting and Scaling DevOps in the Enterprise. I am recommending it very highly to all my customers that are following DevOps! I think its unique – by far the best I’ve read so far when it comes to putting together specific metrics and the questions you’ll need to know in setting your priorities.

Gary notes that there are three types of work in an enterprise:

  1. New work – Creating new features or integrating/building new applications
    1. new work can’t be optimized (too much in flux)
    2. Best you can hope for here is to improve the feedback loop so you’re not wasting time polishing features that are not needed (50%+ in most orgs!)
  2. Triage – finding the source of defects and resolving
    1. Here DevOps can help by improving level of automation. Smaller batch sizes means fewer changes to sort through when bugs crop up.
  3. Repetitive – provisioning environments, building, testing, configuring the database or firewall, etc.
    1. More frequent runs, smaller batches, feedback loop improved. All the DevOps magic really happens in #2 and #3 above as these are the most repetitive tasks.

Notice of the three types above – the issues could be in one of five places:

  1. Development
    1. Common pain point here is Waterfall planning – i.e. requirements inventory and a bloated, aging inventory)
  2. Building Test Environments
    1. Procurement hassles across server, storage, networking, firewall. Lengthy handoffs between each of these teams and differing priorities.
    2. Horror story – 250 days for one company to attempt to host a “Hello World” app. It took them just 2 hours on AWS!
  3. Testing and Fixing Defects – typically QA led
    1. Issues here with repeatability of results (i.e. false positives caused by the test harness, environment, or deployment process)
    2. Often the greatest pain point, due to reliance on manual tests causing lengthy multi-week test cycles, and the time it takes to fix the defects discovered.
  4. Production Deployment – large, cross org effort led by Ops
  5. Monitoring and Operations

The points above are why you can’t just copy the rituals from one org to another. For any given company, your pain points could be different.


So, how do we identify the exact issue with YOUR specific company?

  1. Development (i.e. Requirements)
    1. Metrics:
      1. What % of time is spent in planning and documenting requirements?
      2. How many man-hours of development work are currently in the inventory for all applications?
      3. What % of delivered features are being used by customers and fit the expected results?
    2. An important note here – organizations often commit 100% of dev resources to address work each sprint. This is terrible as a practice and means that the development teams are too busy meeting preset commitments to respond to changes in the marketplace or discoveries during development. The need here is for education – to tell the business to be reasonable in what they expect, and how to shape requirements so they are actual minimum functionality needed to support their business decisions. (Avoid requirements bloat due to overzealous business analysts/PM’s for example!)

  1. Provisioning environments
    1. Metrics:
      1. How much time does it take to provision environments (on avg)
      2. How many environments are requested per month/sprint
      3. % of time these environments require manual fixing before they are complete
      4. % of defects associated with non-code – i.e. environments, deployments, data layer, etc.
    2. The solution here for provisioning pinch points is infrastructure as code. Here there is no shortcut other than developers and IT/operations working together to build a working set of scripts to recreate environments and maintaining them jointly. This helps with triage as changes to environments now show up clearly in source control, and prevents DEV-QA-STG-PROD anomalies as it limits variances between environments.
    3. It’s critical here for Dev and Ops to use the same tool to identify and fix issues. Otherwise strong us vs them backlash and friction.
    4. This requires the organization to have a strong investment in tooling and think about their approach – esp with simulators/emulators for companies doing embedded development.

  1. Testing
    1. Metrics
      1. What is the time it takes to run a full set of tests?
      2. How repeatable are these? (i.e. what’s the % of false errors)
      3. What % of defects are found with testing (either manual, automated, or unit testing)
      4. What is the time it takes to approve a release?
      5. What’s the frequency of releases?
    2. In many organizations this is the most frequent bottleneck – the absurd amount of time it takes to complete a round of tests with a reasonable expectation the release will work as designed. These tests must run in hours, not days.
    3. You must choose a well-designed automation framework.
    4. Development is going to have to change their practices so the code they write is testable. And they’ll need to commit to making build stability a top priority – bugs are equal in priority (if not higher than) tasks/new features.
    5. This is the logical place to start for most organizations. Don’t just write a bunch of automated tests – instead just a few automated Build Acceptance Tests that will provide a base level of stability. Watch these carefully.
      1. If the tests reveal mostly issues with the testing harness, tweak the framework.
      2. If the tests are finding mostly infrastructure anomalies, you’ll need to create a set of post-deployment tests to check on the environments BEFORE you run your gated coding acceptance test. (i.e. fix the issues you have with provisioning, above).
      3. If you’re finding coding issues or anomalies – congrats, you’re in the sweet spot now!
    6. Horror story here – one company boasted of thousands of automated tests. However, these were found to not be stable, maintainable, and had to be junked.
    7. Improve and augment over time these BATs so your trunk quality gradually moves closer to release in terms of near-produciton quality.
      1. Issue – what about that “hot” project needed by the business (which generally arrives with a very low level of quality due to high pressure?
        1. Here the code absolutely should be folded into the release, but not exposed to the customer until it fits the new definition of done: “All the stories are signed off, automated testing in place and passing, and no known open defects.”

  1. Release to Production
    1. If a test cycle takes 6 weeks to run, and management approval takes one day – improving this part just isn’t worth it. But if you’re trying to do multiple test cycles a week and this is the bottleneck, absolutely address this with managers that are lagging in their approval or otherwise not trusting the gated testing you’re doing.
    2. Metrics
      1. Time and effort to release to production
      2. Number of issues found categorized by source (code, environment, deployment process, data, etc)
      3. Number of issues total found in production
      4. MTTR – mean time to restore service
      5. # of green builds a day
      6. Time to recover from a red build
      7. % of features requiring rework before acceptance
      8. Amt of effort to integrate code from the developers into a buildable release
    3. For #1-4 – Two areas that can help here are feature toggling (which you’ll be using anyway), and canary releases where key pieces of new functionality are turned on for a subset of users to “test in production.”
    4. For #5-6 – here Continuous Integration is the healer. This is where you avoid branching by versioning your services (and even the database – see Refactoring Databases book by Scott)
    5. For #7-8 – If you’re facing a lot of static here likely a scrum/agile coach will help significantly.


So – how to win, once you’ve identified the pain points? You begin by partitioning the issue:

  • Break off pieces that are tightly coupled versus not developed/tested/deployed as a unit. (i.e. HR or Purchasing processes)
  • Segment these into business critical and non-business critical.
  • Split these into tightly coupled monoliths with common code sharing requirements vs microservices (small, independent teams a la Amazon). The reality is – in most enterprises there’s very valid reasons why these applications were built the way they are, You can’t ignore this complexity, much as we’d like to say “microservices everywhere!”

I really admire Gary’s very pragmatic approach as it doesn’t try to accomplish large, difficult things all at once but it focuses on winnable wars at a company’s true pain points. Instead of trying to force large, tightly coupled organizations to work likely loosely coupled orgs – you need to understand the complex systems and determine together how to release code more frequently without sacrificing quality. Convince these teams of DevOps principles.


WebDeploy again and IIS setup issues


Went thru setting up a new server with Web Deploy. once again (and it feels like I’ve posted on this a dozen times, but it’s probably only about 10) I walked thru the following:

  1. Remote desktop onto the box.
  2. Setup Google Chrome so I could get around IE’s insane “trusted sites” security-through-obstruction policies.
  3. Installed Microsoft Web Platform Installer 4.6.
  4. Used this tool to install Web Deploy 3.5.
  5. Used WPI to set up IIS Recommended Configuration

At this point I was able to go into Visual Studio and deploy a website out to the Default Web Site\{MySiteName} folder that I created. Yay! IIS_IUSRS had sufficient privileges, etc.

Still though I was getting this error message:

Which was odd.The “ASP.NET is not installed” note is a dead giveaway. My first thought was that we were missing some security settings or a key component. But no, everything looked jake:


I tried an admin command prompt and reregistering aspnet_regiis as follows:

c:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_regiis.exe -i


… but had forgotten that in Windows Server 2012 this is done with the “Turn Windows Features On and Off” app. So, I ran that – and lo and behold, a few key things that ASP.NET 4.5 needs weren’t set up initially:


I guess this is what happens when someone else “sets up” a server partially (sigh). Easy to fix though. After the setup of these missing roles, voila – suddenly I could (without a restart even!) browse to my site locally, and remotely. Easy!

More WebDeploy adventures in Visual Studio…

Here’s a more coherent chain of events for setting up WebDeploy:

  1. Set up IIS (in Windows 8 this is done thru the “Add roles and features” in the Dashboard part of Server Manager. Very nice!) Also make sure ALL the features for .NET Framework 4.5 are set up.
    1. While you’re there, use Search to find inetmgr and add it to your Start/taskbar.
  2. Open up IIS and R-click once you expand Sites\Default Web Site to the wwwroot folder (typically c:\inetpub\wwwroot).
    1. Create a folder named after your website.
    2. Back in IIS, R-click on the folder name under Default Web Site and select Convert to Application.
  3. Note – it’s at this point that I hit an issue with the overly strict security settings in IE on the Windows Server 2012 QA/PROD boxes. So, to get around this, we abandon crappy IE and cut over to Google!
    1. Open up IE tools menu and add and to the list of Trusted Sites.
    2. Go to and download Chrome. Ah, THAT’S better! So long IE!!!!
  4. Download the Web Platform Installer (this should be a part of IIS; it usually will prompt you to run it if you open up inetmgr from scratch).
  5. In the Web Platform Installer I made sure – in the Frameworks tab – that the ASP.NET MVC 4 was set up (along with .NET Framework 4.5):

  6. I also made sure that the IIS Recommended Configuration and the Web Deploy 3.5 options were set up:

  7. If you’re prompted above for Web Deploy 3.5, the Typical configuration is fine.
  8. Time to doublecheck. Open up Control Panel, Administrative Tools – and make sure that Web Deployment Agent Service is running.
  9. If you open up your new {websitename} folder under wwwroot, you’ll also see that IIS_IUSRS for the local machine has read/write privileges. Excellent!
  10. Now back into Visual Studio, and select Publish. Give your profile a name…


The Databases section is interesting – here’s where you can replace on publish your connection strings with the correct database entry for QA /PROD, and test connection strings.


  • Note that if you’ve set up IIS before installing ASP.NET Framework 4.5, you MAY need to run aspnet_regiis to register the framework with IIS. (I didn’t have to do that). If you come across this issue, check some of the notes here.
  • You also want to make sure in the Settings section that you select the checkmark to Delete Additional Files upon Publish. In old-school x-copy deployment terms, this means the same thing as deleting the destination (target) folder before you copy over files.

Found some good notes here on setting up IIS on a completely blank IIS server: It really is (relatively) painless compared to what it used to be; typically your issues will be insufficient permissions on the local machine/network to handle remote connections.


Web Deploy hijinks

I always used to do a file system x-deploy to copy a web app from local out to QA/PROD. But that involved manually changing the web.config and is fraught with manual steps (i.e. errors). So, if you’re getting errors on running deployments, try the following to access Web Deploy 2.0

  1. Remote desktop onto the target IIS webserver
  2. Go into IIS Manager
  3. Right click on the site, and select Deploy, “Configure Web Deploy Publishing”
  4. Click on Setup – > it will create a PublishSettings file locally on the desktop. Copy this to your local system and use this in Visual Studio (manage publish settings in Publish)

Web deployment notes:

From this article – find out if your settings are correct:

  • Click on the Web Site in IIS – select Features tab.
  • IIS Manager permissions icon.
  • IIS Manager Permissions, Action, Allow User – and add your user name.