The Equifax leak, and what you need to do about it.

Perhaps you are – let’s face it you likely ARE – impacted by the leaked consumer information by Equifax. Or the fact that the contact information they give us goes to a bogus number with no viable help offered. And that if you go to their site for remediation you are opting out of arbitration.

One article said – “In retrospect, we find it surprising that it wasn’t multi-trillion lawsuit in light of the galactic stupidity exhibited by a company whose server apparently had zero firewalls from the internet and where any hacker could get access to the most confidential information available….” One commentator put it, “In retrospect it seems like a really dumb idea to give three random companies access to the entire financial records of every American.”

Let’s forget about what a hellhole these huge companies send you to with their “customer service” lines, which are very well documented, or how evil it is that the executives sold their stock in Equifax before news of this leak – which was deliberately kept secret – got out.

If you want to lock down or freeze your credit – and that seems to be the consensus recommendation out there – you can. Below are the four agencies you’ll need to get in touch with. (I put site links in there even for hollow/404 sites just in case they become available later. Right now at least a few are not functioning well, likely due to high call volume and general jerkiness.)

Mail These Two:

  1. Experian – (note this didn’t work, I had to send it in via mail)
  2. Equifax – (didn’t work, I tried 1-800-684-1111 and got a busy signal. 1-888-766-0008, “system error”. 866-447-7559 – 1 – this is just a receptionist that will give you the Equifax info. In short, mail it in. )

Call This One:

  1. TransUnion – (didn’t work, had to call in to 1-800-680-7289 and go thru steps for credit freeze)

Yay, A Functioning Website!

  1. Innovis – (worked, free)


You’ll note, two of the four above required mailings. That’s actually OK for us as we want as much documentation as possible for safety and/or a big ol’ class action lawsuit. Be particularly careful with TransUnion as they’ll want you to sign up for their service which – surprise! – will charge you $19 a month for eternity.

Hope this helped. Now I have to cancel my TransUnion “service” (which I never asked for) thru my credit card, manually. I also went to this site to request a free credit report (you get one annually) from 3 credit agencies.

I need to send another letter to Equifax, on the opt out. I want to sue these guys, VERY badly.

Experian Form Letter – Opt Out

Equifax Consumer Services LLC, Attn.: Arbitration Opt-Out
P.O. Box 105496
Atlanta, GA 30348

The letter needs to include your name, address, and Equifax User ID, as well as “a clear statement that you do not wish to resolve disputes with Equifax through arbitration.”


Experian Form Letter – Credit Freeze

Here’s a form letter you can use for Experian:

Experian Security Freeze

P.O. Box 9554

Allen, TX 75013.




Please put a security freeze on my information.




  • Full Name: ____________________
  • SSN: ________________
  • DOB: ________________
  • Current Address: ________________
  • Previous Address: ________________ (if less than 2 yrs above)


Enclosed: $10 payment (if in Oregon), copy of drivers license, copy of utility bill / bank statement


Equifax Form Letter – Credit Freeze

Equifax Security Freeze
P.O. Box 105788
Atlanta, Georgia  30348


Please freeze my credit.

  • Full Name: ____________________
  • SSN: ________________
  • DOB: ________________
  • Current Address: ________________
  • Previous Address: ________________ (if less than 2 yrs above)


Enclosed: $10 payment (if in Oregon), copy of drivers license, copy of utility bill / bank statement

For the amount you’ll need to pay, see this site: What are the security freeze fees in my state? 

See “Acceptable Forms of Identification for Verification“.



Practical Microservices

Hey all, I am prepping currently a talk on Microservices. Here’s the references I used (very heavily) as sources. I owe these thought leaders a great debt, as they’ve experienced many of the mistakes and pitfalls that can doom a migration to this challenging but enticing model. You must, MUST read at least the first three sources as they’re definitive. And maybe finish up with Jez Humble’s seminal “Continuous Development”?

Thanks as well to Joe Dunn from Marquam for inviting me to be a speaker. Joe, I owe you one buddy, what an honor. Thank you!!!

The Big Three

  1. First is the best IMHO and it sets the tone for everything that follows. Martin Fowler, – essential reading. Includes prerequisites, stumbling blocks, positive aspects of the design. I love Martin, he’s real world but can see the big picture and has lots of experience. The one page omnibus is here:
  2. For governance and a very pragmatic balance between safety and velocity with your architectural reviews, “Production Ready Microservices” by Fowler. Microservice evaluation is organized by topic. This is real world and based on her experiences at Uber. Thoughts on alerts and dashboarding are solid and well thought out. You can crib from this yourself in building a checklist for your audits.
  3. How micro is micro? A great discussion on Domain Driven Design, chapter 5 of “Microservice Architecture Aligning Principles, Practice, and Culture” by Nadareishvili et al – Microservice Architecture. It includes a great practical breakdown of handling one workstream and defining service boundaries using DDD of a sample company.

Secondary Sources

  1. How Soundcloud migrated from their monolith using the strangler fig pattern – three parts, first one is the best IMHO – Phil Calcado: (part 2 is here, part 3 is here)
  2. How complex architectures like Netflix are in reality –
  3. For an end to end Microsoft implementation Bob Familiar’s book on “Microservices, IOT and Azure” is very solid. I wish I would have had more time to include his walkthroughs on containerization with Docker.
  4. Building Microservices by Sam Newman. Again, time didn’t allow me to use his material much, but it is VERY valuable here, and very explicit/tactical in its approach. I love it!
  5. Reactive Microservices Architecture – free ebook, design principles, very Java based. This seems to tie in very well with M Fowler’s thoughts on designing for failure.
  6. Microservices with Docker on Microsoft Azure, by Scholl / Swanson / Fernandez.
  7. Video from Chicago in 2015 – “How I Learned to Stop Worrying and Love Conway’s Law“. James Lewis. I love this one because there’s a few examples where they knew the org was not capable of the change needed – and designed a system that would fit it (square peg in square hole!) instead of dictating how the design should work in a perfect, idealistic world.
  8. Zhamak Dehghani – Real World Microservices video – holy smokes this is old, all the way back to 2014!
  9. Randy Shoup on Microservices and Conways Law (video and transcript)
  10. Implementing Domain Driven Design, by Vaughn Vernon. Decomposition and finding domain boundaries.
  11. Video by Sam Newman, outstanding.
  12. A free ebook on this page as a PDF – excellent resource for Docker containers and .NET architecture. The ebook is here. The same page has a ref to a “Containerized Docker Application Lifecycle with Microsoft Platform and Tools’ book that looks excellent.
  13. Yes there are ways to make transactions work in a distributed system. It just takes some work, but there are some usable patterns out there. (I think it’s hard to argue that it will be harder to debug and implement than with a monolith however!) Here’s one article on the subject, with a nice breakdown of DDD to boot.


The Cheat Sheet – Who, What, When, Why.

What are Microservices?

There’s a few definitions out there, which seem to complement each other.

Short version –

  • “A software building block that does one thing and does it well.”
  • “Microservices are small, autonomous services that work together.” – Sam Newman, Thoughtworks Adrian Cockcroft, “Loosely coupled service-oriented architecture with bounded contexts.”
  • Ideal for big systems (new problems arise because of their scale – beyond boundaries set by initial architects.”

The upshot is that this is a design that values replaceability over maintainability – i.e. cattle not pets. Architecturally it borrows heavily from SAAS, Continuous Delivery, Agile, and DevOps. And yes even SOA!

What are the benefits of moving to microservices?

There’s actually a lot of benefits with this type of architecture – and unlike SOA there’s a few real world success stories we can point to (Netflix, Amazon, and Microsoft etc) that all decreased release sizes and improved maintainability using a microservice architecture. It allows technical diversity (I can write my code any way I want using different libraries as long as the API layer is consistent), independent deployments (a huge win, suddenly I am unshackled and able to get my bits out the door faster without waiting on tightly coupled partners and huge testing cycles), open standards, and flexibility.

Uh huh, I’ve heard that before. What’s the catch?

Microservices aren’t a silver bullet and anyone telling you it is is trying to sell you something! They are much, much harder and more expensive to monitor and control. They seem to be a better fit for large complex systems, at companies that stream content and are otherwise loaded up with talented staff with a high level of proficiency both in the business domain and in DDD theory. It’s hard to prototype Microservices too – one quote that sticks out in my mind is, “the decision to move to microservices must be company wide.” Agile has been very successful as it is internal to one team; DevOps less so as it requires cross functional groups. Microservices, with an incredibly complex and hard to troubleshoot operating environment that is constantly in flux and rarely consistent (i.e. a debugging nightmare), will often make your siloes of knowledge worse not better and if you’re not prepared to enforce governance it can make maintenance worse not better and create a spiraling technical debt issue as teams work in isolation without considering quality or consistency – the exact opposite result you’d expect with this type of architecture. Again, see Susan Fowler’s terrific book on “Production Ready Microservices” for a roadmap on enforcing consistency for more on this.

I love it. What do I need to do to get here?

Well, you’re going to need to do a health check on your readiness before you make the leap to microservices. How long does it take you to provision an environment? Do you have a templated set of builds that are in source control – i.e. infrastructure as code? How capable is your monitoring? (You’re not going to have a lot of time to track down issues, alerts must be actionable and real-world and your devs onboard with handling support 24×7). How automated is your release process, and can you run deployments out to production – yes gated at QA – with a high level of certainty on quality? (This implies a very well thought out and automated test strategy. You can and should think feature flags, canary testing, etc – but the tighter the cycle from dev to prod and the more automation, the better your ecosystem can handle the leap to MS.) Do you have a top down commitment to quality – are your executives/business stakeholders comfortable with partnering with you and viewing quality as a worthy investment? Last – how far along are you on your DevOps journey? Can you form cross functional teams or is there a wide gulf between you and your siloed IT/Ops partners?

Eeeeshhhh. We’re just not there yet.

Good! That’s a sane reaction and it’s a very good thing to look before you leap. Microservices are awesome but still very new and we’ve already seen enough to know that they will not work in just any organization. (thank you Conways’ Law!)

That being said, there is much you can do to improve things without making this kind of a wholesale leap. For more, check out this post by Donovan Brown. In essence, he says to start with your pain point. Maybe your agile SDLC is humming right along but your operations team is struggling to keep up. Or maybe it’s taking 3 months to provision dev or test environments, and they don’t match production. Or perhaps your RM process is manual and error prone.

The roadmap that Donovan discusses starts with the following:

  • Source Control
  • SDLC – Kanban boards, accountability to business, unit testing, team definition of done.
  • Automated deployment to dev
  • Automated testing – automated unit tests gating out to QA, not PROD we’re not crazy, and a sane level of integration testing
  • Infrastructure as code – no more hand coded or tweaked environments, we’re using templates checked into source control to enforce consistency. We’re not patching, we’re replacing!
  • Monitoring – no more drowning in a barrage of alerts, but actionable alerts that give us a heads up before availability issues arise
  • …. And rinse and repeat. Tie this into your business values and try to show KPI’s so you know if your DevOps improvements are gaining traction or if you are just spinning your wheels.

That’s all folks….

I will continue to update this. And I’d value your feedback!


DevOps – how we do it at Microsoft, with Aaron Bjork.

Two great interviews I found today and enjoyed very much, between Donovan Brown and Aaron Bjork (who is head of the DevOps part of VSTS).

From the first video:

  • We treat our development teams as adults – “I’ll trust you until you give me reason not to.”
  • How does Microsoft structure their product teams?
  • How do we keep to a consistent UI? i.e. checkboxes versus radio buttons – what design guidelines do we use?
  • How to tie together 50 teams with one weekly meeting of six people
  • How big of an advantage it is to dogfood it – be your own first customer.
  • “Our engineers write code, test code, and deploy code.”
  • Feature flags – we don’t use these for just one customer. The boundary to turn on the flag is during the current sprint, and then turn off the following sprint.

Second video notes:

  • (minutes into program) 1:27 – Why 3 week sprints? Goldilocks principle. Donovan mentions a physician product owner where 4 weeks was the best he could get.
  • 5:46 – Dogfooding, and customer driven features.
  • 13:09 – We used to have all these plans that helped me sleep at night – but did we ever ship on time? (i.e. waterfall and requirements planning as a security blanket)
  • 16:32 – importance of monitoring. DevOps is not automation. Most of their DevOps discussions end with Agile.
  • 19:03 – distrust between Dev and Ops. The chasm grows. Now we are improving with TDD, unit testing – but the distrust is still there.
  • 20:27 – treat your deployment pipeline as a feature of your product.
  • 27:54 – Do we deliver mid-sprint at Msft?
  • 32:57 – “You can’t cheat shipping.” Scenarios, features, stories and tasks.
  • 39:10 – Feature flag litter and documentation.

DevOps – are you stuck on where to start? Building out 4 Release Pipelines in an Hour with Donovan Brown

Such, SUCH a great treat having Donovan come all the way from Houston for this presentation. He’s a fantastic presenter and his hands-on knowledge blows my socks off. I can’t thank him enough for making the trip.

IF YOU ATTENDED and you enjoyed the presentation – or even if you didn’t enjoy it – WE NEED YOUR HELP!
Take a few minutes and click on this link – it will tell us how we can improve next time, and hopefully lay the groundwork for Donovan to come back in a year or so.

There’s a streaming video available here of Donovan’s Portland Area .NET User Group (PADNUG) presentation. It was outstanding! If you want to see one CI/CD release pipeline built out releasing a website to Azure in one hour – well sorry, that is not what he did. Instead, Donovan built out FOUR pipelines in an hour – using Node.js, Java, .Net and .Net Core., while describing step by step what he was doing and why – using VSTS, the Azure Portal, Visual Studio, and a cool command-line tool I hadn’t heard about called Yo Team!

The key takeaways from his presentation are as follows:

  • If you only get one thing from our time together – Microsoft is for any language, any platform.
  • Licensing costs should not be a blocker for us. It’s like $10/person per month, and that’s only for large teams over 5. And with MSDN you get these benefits included.
  • Lastly, if you attended and enjoyed the presentation – give a shout out to Donovan on Twitter. He’s thrown down the gauntlet – “we have the best build and release tools on the market, if you don’t agree reach out to me on Twitter and be specific!” So, if you don’t think Microsoft’s build tools will work with Jenkins, or Java apps, Node.js, SonarQube etc – throw down and enter the squared circle my man!

My notes are below. These aren’t notes of his presentation as they were no-slide demos, but comments Donovan made during the day that gives DevOps context to his presentation. As the saying goes, “If I had more time it would have been shorter!”




Thoughts on Strategy

Make incremental changes in ramping up your maturity. It’s safer. If you are deploying manually, there’s one part that you fear the most. Focus on that.

“If it hurts, do it more often.” (You are incurring more risk by deploying less frequently.) Lean forward into that and think about, how can I fix this. Focus on scariest part first.

If you need prescriptive – hire a consultant. (Like, I don’t know, Microsoft Premier for Developers maybe!) They can ask questions, pinpoint pain points for your org and provide a specific roadmap.

Why do you have to ask permission from your manager? Your job as an engineer is to continuously deliver value. You should need to ask permission if you DON’T want to set up CI. No need to stop and learn for months – focus on what hurts the most, and focus on that one thing. What is the thing – every time you run a deployment – why it breaks. Otherwise – you end up paying a tax doing it manually. Find something small to focus on – don’t try to do it all at once. It might take you months – after which you may have an amazing portion of your pipeline that was manual that is now automated. That will get people excited. And after that last build that blows up when people start playing the “Not it” game – guess what, you get to go home early!

And if you’re stuck on exactly where do we start? – (really, check this post out. It’s good enough to mention twice!)


Agile Is Your Starting Place

The key question is – “Can you produce releasable bits of code?” Until your Agile processes are strong – not perfect but mature – you can’t see significant gains with DevOps. The business needs to understand that we need to produce increments of shippable code. DevOps is the “second decade of Agile.” Once Agile is in place you will realize that you’re not able to ship as fast as you can produce features.

What limits us so far – four years after “The Phoenix Project?” Really it’s still slowed by adoption of Agile. 40% of orgs have adopted Aguke. That’s a dead number, as its only dev teams that are championing this. The rest of the org is expecting waterfall, requirements, dates.

DBA’s are frequent blocking point. If you slice vertically and not horizontally. I’ll give you 3 months, if you promise not to make any further changes to the schema, ever. “That’s crazy!” “If you’re going to change it anyway why should I invest 3 months in this now. Your first step could start with login functionality – in 3 weeks we have a login page, username and credentials, authentication and identification.” Don’t let a recalcitrant DBA stop you in your tracks.

One week sprints really worked well (at one company) for Donovan in his consulting past- with product owner/business owner. After 8 weeks, needed to get a knowledgeable person (replacing old business owner) – and a sigh of relief, finally back on track. And hire a scrum master. That person needs to be able to say no to the boss. For a good product owner, all they have to do is understand the business they are trying to serve – and tell me of two backlog items which one is a priority. Scrum Masters must protect team from unrealistic expectations, on this date, I want these features – perfect quality. Quality is not negotiable – either move date or lose features.


Database Goodies

SQL Server Data Tools (SSDT) – version every object on SQL Server including permissions and roles. Stored in a database project. (this is free. Evaluate, see if it meets your need.) An amazing tool, been around for 10 years, very little known – our best kept secret! If SSDT for some reason isn’t scaling or working for you – ReadyRoll from RedGate is also a good option.


Last but not least – this amazing site brought to my attention from Jorge Segarra. An awesome resource for all things DevOps for you DBA guys!

This may require breaking up into diff schemas (like a virtual namespace) vs one huge project. Schemas are here now and available; will allow you to segment databases that you create – no 3000 tables/etc in one untranslatable blob. (BACPAC is schema and data, DACPAC is just schema. Can run post/pre scripts- make sure its idempotent so if it runs twice you don’t double your db size!)

Migrating your CRUD statements from sprocs to Entity Framework may be a win to move away from thousands of unwieldy and clunky INS/UPD/Del type statements.


Integration Testing (Selenium etc)

DevOps – one company said “DevOps = Automation”. So they had thousands unit and integration tests. Code would flow with no human intervention from commit to production. As soon as they release a CSS change though to prod, they start losing money. As soon as they started looking at ecommerce website – summary page made final buttons and the text the same color- this resulted in a blank button! Automation clicked it because the ID of the button did not change. So, automate everything you CAN – don’t try to automate things you shouldn’t.

Best practices for building integration tests: the #1 thing is making sure assumptions you made when you recorded your automation are still valid. Usually the database in QA is the failing point: the code isn’t broken but the data it runs against doesn’t fit assumptions. See Redgate’s excellent tool SqlClone here.) They are a few versions behind with their Selenium driver, using IE.

“We spend more and more time with peer testing than we do with auto testing on the Microsoft VS team. As we’re our own first customer, there’s a lot of A/B testing that happens. Integration testing happens very little, unless there’s a performance issue – then we move it to GoBig environment where we can perf test.” Per Brian Harry – “Unapologetically we do testing in production.”

Donovan ran a team in Akron where they were not allowed to write a line of code until they wrote a UI test (in CodedUI, but now we do this in Selenium). They wrote automation before they wrote code – wrote so it fails, then refine until it passes. This and a manual test case – which can be given to a stakeholder in plain English (“oh, I didn’t expect I should be pressing that button”) – there’s nothing better than a manual test case to be your new acceptance criteria. Automation and UI tests are brittle – they become brittle/break, that didn’t go away, but having it there forced my teams to do the right thing early. (These were one week sprints, Weds to Weds.) Do it more often if it hurts! Finding clever ways of making BACPAC/data layer work.

VS teams – forced them to move more functionality to behind the scenes, UI testing. Make unit testing viable. To those nerd architects out there claiming that “Unit tests are worthless”! – we run 41000 unit tests in VSTS and it takes 6 minutes. You can’t run integration tests that fast.


Infrastructure as Code (ARM templates) and Configuration Management (PowerShell DSC)

ARM (Azure Resource Manager) is the name of the game here to get Infrastructure as Code working. This is a much better process than spinning up resources manually for example. (Not configuration here – doesn’t include IIS, etc. )

Recommendation: Introduce your Infra team to ARM templates. ARM templates are Azure only but you can use Terraform to point to Azure/multi-cloud solution so you can move resources to Google/AWS etc.

Configuration as Code – using PowerShell DSC (Desired State Configuration) – take configuration of that server and you codify it as well. (I demand you have port 80 open, .NET 4.5, IIS available, these security roles etc.) Server installs everything you need to bring it to ready state. Every 15 minutes – checks to make sure it fits standard config. So, if it finds that IIS is turned off, it will detect if the configuration is drifting and make a note in the log. You can also configure it to bring it back to standard state.

PowerShell DSC works for both Linux and Windows environments. And, it’s free! (p.s. Chef uses DSC as well so it’s a layer on top of a layer.)

Recommendation: Start with what you have and what’s free with PowerShell DSC and THEN make an evaluation if there’s features that are missing. This could take <1 hour. From Donovan – “I encourage you to ping me on Twitter if DSC is not getting the job done. If we can’t fix it fast enough, then go to Chef or a competitor.”

DSC can be done onprem/etc. It’s idempotent, which means you can run it blindly a million times in a row and the server will stay exactly the way it needs to.



Self Service VM’s with Dev/Test Labs

Struggling with self service VM’s and exposing costs of resources to the consumers you service? (I.E. that 5 9’s coming at a horrendous price that is shielded from end users/business?) Think about Dev/Test Lab. This is Self service – no blank check, you can spin up a sandbox (using allocated $ your customers buy each month) and enforces the VM images they are allowed to spin up.

Dev/Test Lab documentation –

Overview and trial site hub –

Starting documentation, very good –



Of advantage because releases now are micro sizes. No more having to track down 6 months back with a dictionary of changes. Would eliminate webmethods – ESB. Can rev very quickly on microservices without jeopardizing the application as its very small, atomic pieces of functionality.

Meant to be combined with containers.

Not a silver bullet. Evaluate it. Microsoft is going there on VSTS team. Docker support is built into VSTS – can publish to a registry. Even TFS 2015 update 3-4 has Docker support as well. Need to also run Win Server 2016 to support containers, or Linux containers. Anniversary edition of Windows 10.


A few thoughts on success.

Hey all, it’s been awhile so I thought I’d write a few words.

I had someone new to Microsoft call me up today and ask me for some thoughts on how to succeed here. I hung up after chatting for a few minutes about time management and learning to say no – modesty – and how to work in larger teams. It’s a topic that is interesting to me – as every company has its own dynamic (one that spills over into the technical arena) – and I often find myself wondering if I could improve in this area.

If you haven’t already read the Four Agreements, I would highly recommend it. It’s one of those short books that’s really easy to overlook. Besides the Bible, I can’t think of another book that’s had such a high impact on my life, for the better. If you want to make a successful career though – it’s hard to give better advice than the following:

  • Be Impeccable with Your Word. This means if you say you’re going to be somewhere or do something, you keep your word. It also means you need to be very careful with what you say yes to. A modest person knows they have limitations, and they’ll be careful not to overload themselves. (Something I’ve had to learn the hard way a few times in my recent assignments!)
  • Don’t take anything personally. Everyone is the hero of their own story and their actions – while sometimes senseless to us or worse – always make sense in their own personal universe. Do your best in working with others to be accepting, to listen first and understand (reflect back actively what they’re saying so they know they’re being heard). Once you understand their point of view and motivations, it’s much easier to work towards a compromise that satisfies both parties.
  • Don’t make assumptions. We assume that others think the we think, feel the way we feel, judge the way we judge. The only way to keep ourselves from making assumptions is to not be afraid to ask questions. And we keep asking questions until we understand clearly. Until you know – this is what I want, this is what you want. The unstated becomes clear, no secrets.
  • Always do your best. This means trying your damndest at everything you take on, and doing it without distraction. Now, your “best” is a relative term – it doesn’t mean perfect, and your best will vary from day to day. Some days just getting up and responding to emails is “your best”, on other days you’ll be a worldbeater and knock down walls. Don’t try to solve problems in a low state. We all have ups and downs due to low sleep, stress, personal problems etc. As my good friend Jim Caballero once told me, “people are the only thing that matters”. I’ve rarely regretted the way I’ve started or left a position, but I do regret many times falling short in my interpersonal relationships. That thoughtless exchange of words with a coworker when I was feeling a little down, the meetings that did not go well, etc. Many times when I am low on sleep now I try to avoid one on one meetings – so I don’t have to spend the following day mending fences I’ve broken down!


I’m definitely not holding myself up as an example in any of these things! But I will say – over the past two decades – learning by experience how to apply these four points has made a huge difference in a smoother, less bumpy career and a higher ceiling. It’s been said that we get hired for our abilities – we get fired for our personality. True words! So keeping in mind that old saying “the tongue is a fire” – guard your words, be careful with your words (especially those committed to writing – emails, Facebook, etc), and show empathy. You’ll get more done and will be happier doing so!