I was at a conference earlier this week and we got some outstanding questions about how Microsoft went about their transformation – especially with the Azure DevOps team. I want to build on this with a followup post going into more depth on our use of culture and automation – but here’s a good place to start with some great links.

Question #1 – How do we handle planning on a strategic level with the more tactical focus of Agile?

• Using features and epics – I love this – https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/define-features-epics?view=vsts&tabs=new-nav
• How we handle epic level planning at Microsoft – Cloud9 Donovan Brown interview with Aaron Bjork, 34 minute video. Excellent.
• Note there’s a relatively new feature in Azure DevOps to report across teams called Delivery Plans. More information on this and how to set up reporting is here. A detailed walkthru on implementation is here.

 
 

Question #2 – How did Microsoft go about their transformation to DevOps from a shared services model?

• See my interview with Aaron Bjork, and the outstanding YouTube video he did in the footer. One of the best 45 min videos on the topic I’ve ever seen – anywhere. https://driftboatdave.com/2018/05/30/devops-stories-aaron-bjork-microsoft/
• A great article on “What is Agile” from Aaron Bjork. https://docs.microsoft.com/en-us/azure/devops/learn/agile/what-is-agile
• Our best DevOps “how and why” articles are by Sam Guckenheimer – great guy – see this central hub for more. https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops

 
 

Question #3 – What about testing? (This is usually one of our biggest blockers to improve release reliability and velocity – an unreliable, flaky test layer)

• Great interview page with Munil – https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/shift-left-make-testing-fast-reliable
• Also on Eliminating Flaky Tests (we want red to mean red) and our unabashed use of testing in production. This caused great angst for our developers but ended up being the single biggest contributor to our success.
• Our branching and release strategy. We did experiment with GitHub Flow but found it didn’t work in our specific case. We do work off of trunk – long-lived feature branches (>1 day) are verboten.
• We like rotating “F” and “L” teams so some of your people are handling direct (livesite) report, others are focused on development. Note that this doesn’t mean 100% of all support calls hit devs directly – it might just be 5% – but devs must share some operational support for DevOps to work. And you want to tune your alerts so if someone is getting woken up at 5 am, there’s a damn good reason – the Google SRE book and “Practical Monitoring” goes into more detail. See Aaron bjork’s overall presentation I mentioned before, or this page for some great videos on our move to a livesite culture.
• We also shifted left on security. Here’s a good walkthru.
• How we moved from a monolith to cloud-based microservices, by Buck Hodges. He says point-blank that moving to the cloud the way MSFT did – lifting and shifting – was a big risk, like jumping off a cliff. Looking back, it ended up being the best way forward.
• Using feature flags and release rings to control the blast radius.

Question #4 – Production Support. Let’s say we have an Agile team, 8-12 people. How the heck are we supposed to do global support across multiple regions, 24x7x365 in production?

• Short answer – the only way this will work is if you 1) make sure you’re only supporting a small sliver of functionality, 2) that you gate the support demands upon your devs so it’s <50% of their time i.e. the SRE model. More than likely you’re going to have some operational support – even offshore or 3rd party – handled externally to the team. 3) alerts are tuned so that only truly important things make it through. I talk about this extensively in my book; the books “The Art of Monitoring” and “Practical Monitoring” also elaborate on this.

 

 
 

Just published a great podcast interview with Betsy Beyer and Stephen Thorne of Google, coauthors of the incredible “Site Reliability Engineering Workbook”. We cover a lot of ground in this interview, including how Google learns from failures, what toil is and how good organizations try to fight it, and when LESS reliability can be a good thing in software development.

Betsy and Stephen’s work around publicizing and breaking down the myths surrounding SRE have been huge boons to our industry. Their writing (along with a few other Googlers) had a big impact on the Achieving DevOps book; we find their work and thinking incredibly helpful and influential. You’ll love it too!

A link to the interview is here – and it’s on the podcast platform of your choice. Apple, Google, Spotify, blah blah…. We’re on all the major platforms now, including Anchor, Apple, Google, Spotify, PocketCasts, and RadioPublic. Please support the podcast, and we’d love to hear your feedback about the book!

Enjoy the podcast!

Some link goodness:

• Link to SRE landing page: https://landing.google.com/sre/books/
  
• SRE workbook online for free: https://landing.google.com/sre/workbook/toc/
  
• On LinkedIn: https://www.linkedin.com/in/betsy-beyer/ | Stephen Thorne
  
• Niall Murphy on Twitter: https://twitter.com/niallm?lang=en
  
• My original interview with them is here
  
• The Google landing site for SRE – https://landing.google.com/sre/book.html – free PDF versions of the revised [sre] text and the followup handbook. This should be must-read material for anyone out there, from IT practitioner to CXO exec.
  
• “Managing Misfortune for Best Results” by Kieran Barry at the SREcon EMEA, https://www.usenix.org/node/218852. This is a great overview of the Wheel of Misfortune exercises in simulating outages for training, and some antipatterns to avoid.
  
• The Google landing page Betsy mentioned with lots of rich content – https://landing.google.com/sre/
  
• Advance copies of Betsy’s next book can be reserved on Amazon. “Building Secure and Reliable Systems: SRE and Security Best Practices”

So thrilled to have Gary as a guest on the Achieving DevOps podcast!

Gary Gruver is one of my personal DevOps heroes and is very well known for his landmark books. Unlike some of the writing I see out there, his writing reveals a solid background in the manufacturing field. He brings a process-driven engineer’s perspective that is always enjoyable and enlightening! He’s worked with HP, Macys, and a host of other enterprises in helping them achieve a stable quality signal and improve their delivery pipelines.

His current passion is helping as many people as possible achieve similar results through consulting, webinars, presentations, and books. Gary’s consulting approach is unique in that he stays engaged with organizations during their journeys to provide guidance when they hit specific challenges and to learn about what is and isn’t working during as many different transformations as possible. This experience enables him to help clients avoid common mistakes and share approaches that have been proven to deliver business results for a variety of companies. He is the author of “Starting and Scaling DevOps in the Enterprise” – a book that had a huge influence on my thinking – and co-author of “Leading the Transformation: Applying Agile and DevOps Principles at Scale” and “A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware”.

Check out Gary’s books on his site below – I know of no one else in our industry that brings what he does to the table when it comes to optimizing flow and eliminating waste and repetitive work. A link to the interview is here – and it’s on the podcast platform of your choice. Apple, Google, Spotify, blah blah…. We’re on all the major platforms now, including Anchor, Apple, Google, Spotify, PocketCasts, and RadioPublic.

Enjoy the podcast!

 

 

Links to enjoy:

• The official Gruver Consulting site: https://garygruver.com/
  
• Gary on Twitter: https://twitter.com/GRUVERGary
  
• LinkedIn: https://www.linkedin.com/in/garygruver
  
• My original review of “Starting and Scaling DevOps“, an incredibly valuable book that I still read from every 6 months or so: https://driftboatdave.com/2016/12/20/starting-and-scaling-devops-in-the-enterprise-review/