HWTSAM–Five Years Later

Five years ago this week, How We Test Software at Microsoft hit the shelves. The book has done well – both in sales, and in telling the story of how Microsoft approaches software testing.

An unfortunate aspect of writing a book like this is that after five years, most of the book is obsolete. Sure, there are some nuggets that stand the test of time, but so much has changed – and is changing, that it’s hard for me to recommend much of it as practicum. I don’t know if a new version is warranted – or necessary, but you never know what will happen.

To be clear, I’m still proud of the book, and think it has a lot of great ideas that will help many companies – it’s just not an accurate reflection of how Microsoft tests software. Here’s a section by section recap / overview of what’s changed….

The New World

Section 1 – About the Company

Now that we’re a Devices and Services company, the description of our product groups is flat out wrong. Beyond that, our organizations are much different – our test dev ratios are changing (from nearly 1:1 at the time of writing to nearly 2:1 dev:test today).

Five years ago, a few teams used adaptive (agile) engineering approaches, whereas now, nearly every team in the company is using or experimenting with some flavor of adaptive development. We ship almost everything faster. Testers are moving to analysis of end-to-end scenarios and data analysis, while “traditional” functional testing is moving (where it belongs), to the programmers. The career path examples we showed in the book have all changed – and will likely change again as the company continues to adapt and grow.

Section 2 – About Testing

The techniques described in the about testing section are all still relevant (although these days, I could easily increase the chapter on test design by about 50 pages and keep it full of good ideas). What’s changed most (and what will continue to change) is how much of the activities in this section are carried out by developers (and how much more of will be carried out by the developers in the coming years.

I’m still particularly happy with the chapter on Model-Based Testing (and thanks to Michael Corning and Harry Robinson for their help with the ideas for this chapter).

Section 3 – Test Tools and Systems

The section on bug reports is good – as long as you have the need to track bugs formally. Many teams can (and do) get by with fixing bugs as they find them – with no need to document the bug. I’ve also become even less of a fan of tracking test cases, but the examples are probably still valuable for those who need to do this.

It’s hard to believe that five years ago, I failed to recognize the critical relationship between test automation and test design (they’re in different chapters!). Although the technical information is correct, I would like to take The A Word and shove it into chapter 10 as an addendum.

The customer feedback systems I wrote about have expanded massively – and are much more of a focal point these days – both in the data we gather, and how we use it.

And finally (for this section), what blows me away is how much different our approach is to testing web services. I don’t think Ken will be mad if reveal that this chapter is embarrassingly obsolete. This isn’t because what we were doing then was bad…it’s just that we’ve grown so much in our approaches here that our five-year old processes seem obsolete in comparison.

Section 4 – About the Future

I suppose this section is still relevant  – and most of the future as I saw it in 2008 is reality today (although much of this section, in hindsight, could have been better written.


In a perfect world, I’d write another HWTSAM. Maybe someday, I will, but it’s not on the roadmap for today. There are too many big changes coming (both planned by MS, and spinning in my head) for me to put a stake in the ground. Or maybe, software is changing so much that trying to capture how a company does it at a point in time is impossible. I’ll need to think about it.

For those of you who bought (and read) HWTSAM – thank you. There’s still some good stuff there, but read it with a grain of salt (or at least from a lens knowledgeable that it’s a half-decade old book).

Book Stuff on Leadership

I intended for my last two posts to cover the main points of my STAR West keynote, but I neglected to mention a few of the books I referenced in my talk (which not coincidentally, are some of my most recommended books for those studying leadership.


I first read Leadership on the Line as part of a class I took several years ago, and many of the concepts have stuck with me. My favorite quote (slightly paraphrased) is, “Leadership is disappointing people at a level they can tolerate [absorb]”. Too many people try to lead by pleasing everyone. Good leaders are comfortable making decisions that not everyone agrees with, and know that teams are more productive when there is a slight level of discomfort or apprehension (as opposed to the extremes of complacent or overwhelmed).


I am a huge Patrick Lencioni fan. Lencioni writes business novels – stories about business that explain some of the principles and ideas he believes in. His Getting Naked is a peek into how he runs his consulting business, and mirrors the way I approach leadership and influence. Humility and learning in context are great approaches for anyone diving in to a new project, and I have found every one of Lencioni’s novels to be a page turner. Lencioni (in his one non-business novel publication) is the first person to clue me in on the fact that product quality is directly related to team health – something I’ve been reading more about over the past few years, and something I believe in whole-heartedly.

imageIn my experience, there are a lot of parallels between good leadership, and good consulting (bearing in mind, that the definition of a consultant has changed remarkably over the past several years).

I could write a whole post on how this book relates to leadership. And, in fact, I did (and if you’ve read this far in this post, go read that post now for more info).



Several years ago, I had an opportunity to hear Peter Spiro speak, and to also have dinner with Peter and his daughter while on a business trip (we were both speaking at the same internal conference). In his talk, and at dinner, he recommended The Feiner Points of Leadership. I read the book (and have re-read it several times), and I always seem to find ideas I can use (and ideas that are weirdly relevant to whatever I’m going through). The writing is conversational, and the advice is practical (opposed to the theoretical hand-waving in most leadership books).

There are at least a dozen other leadership books on my bookshelf that I like, but if you were to see my beat-up ragged copies of these books, you would know they’re the most used (and most loaned) books in my personal library.

Stuff About Leadership

In my last post, I wrote a bit about what it takes to build a great team, and how it takes a great team to build great software (which also covers about half of what I spoke about at STAR West in October). In this post, I’ll see if I can cover the other main points from that presentation.

Leadership and Management

Many people still get leadership and management confused. One does not have to be a manager to be a leader (and most certainly, not all managers are good leaders). As someone who has spent the majority of their career as a peer to test managers (yet in a non-manager) role, I frequently see the manager role as one that runs the testing business, while my role is to improve the testing business (in fact, I would say that many of my non-management Principal band peers provide as much, or more leadership to the team than the people-managers on the team).

Many people (and company cultures) assume that career growth for a software engineer always involves a “promotion” into management. Too often, this policy results in turning great software leaders into mediocre managers – or worse, enables mediocre engineers to prolong their career by becoming completely inadequate people-managers.

The formula to fix this isn’t difficult.

  1. Let the people on your team with demonstrated skills in people management manage the team (and there are dozens of ways to discover who has, and does not have these skills before making someone a manager)
  2. Encourage strong engineers to take on team leadership (but not necessarily management roles).

In my last post, I said there’s a direct relationship between team health and product quality. One surefire way to negatively impact team health (and subsequently product quality) is to put people in roles they’re not ready for – or roles where they cannot succeed.

Learning Leadership

Not many things irk me more than self-proclaimed leaders. Being a great leader involves study and hard work. Just as great engineers know patterns and heuristics that enable them to build great software, great leaders know patterns and heuristics that enable them to lead.

Too many leaders (or those who call themselves leaders) rely on only a few leadership tools – often thinking that if they merely tell people what to do, that they’ll do it. Leadership is influence, and influence requires credibility, compassion, humility, strategy and many other factors. You can’t lead with only a handful of leadership ideas – you need dozens or more.

It’s just as important for a great leader to have a toolbox full of leadership ideas as it is for a software engineer to have a toolbox full of design ideas. Not enough people understand this point.

If you think you’re a leader – or if you want to be one, experience alone is not enough to get you there. Work at it, practice, and learn. And then work at it, practice, and learn some more.

Some Stuff I’ve Learned

I picked up my Xbox One this morning – a special white console with “I made this!” engraved on it. The reviews started appearing last night, and for the post part, they’re quite positive (and I’m not surprised by the drawbacks listed in the reviews I’ve read so far). Now that this project is officially behind me – and before I figure out what’s next, I have some backlog blog posts to share.

Lessons Learned

I gave a keynote at STAR West in October on “Lessons learned from Xbox One”. Given that it was an unreleased product at the time at the time of the presentation, rather than talk about the details, I spoke mostly about the team, and how strong team principles lead to great software projects (I spoke of my love of the Xbox team in a previous post – http://angryweasel.com/blog/?p=723). I thought I’d share a few of those lessons.

Lesson 1: Build a Great Team

In order to build great software, build a great team first. I’ll say that differently in case it didn’t sink in (or make sense). Worry about building a great team first, and then let them build a great product. I realize this isn’t always an option for every organization, but I see too many teams take an existing group of people, and then try to make a product that isn’t in the sweet spot of that organizations strengths. A good team can usually make a few good things, but a great team can make anything great.

That leads to a phrase I heard once that rings too true – “Don’t ship your org chart”. Just over ten years ago, I was working on Windows CE (an embedded operating system loosely based on Windows). During product planning, I noticed what looked like an unbalanced number of networking features (there were a lot – including some relatively obscure functionality). I didn’t know if it was a product strategy decision, customer requests, or what, so I asked. The answer was, “Our networking team has a lot of people, and we need to keep them busy.” To this day, I still don’t know why there was no attempt to shift people around as needed for features customers wanted, but I do know it seemed silly at the time, and it seems silly now.

Which leads to…

Lesson 2: Diversity and strengths

I’m a believer in teams filled with (or at least largely populated by) specializing generalists. Everyone certainly doesn’t have to be able to do everything, but every team member should be able to do many things well, and a few things very well. It’s important, when building a team to have diversity in both generalization and specialization – and match those skills with what’s needed for your project.

In my opinion (and experience), you can’t have enough deep specialization. My official job title at Microsoft is Principal SDET – that means I’m in the Principal band, and that I don’t manage people. I think far too many testers (and organizations) see management as the only career growth path, when there is huge value in developing, growing, and nurturing the experience of experienced individual test contributors (which reminds me, I wrote a paper about the value of highly experienced non-manager testers a few years ago).

There are 100 or so Principal SDETs at Microsoft (plus more in the management ranks that I haven’t counted lately). When I joined the team two years ago, there were five of us (Principal testers) – two on the Xbox Live team, and three on console. For various reasons, we wanted a name for our virtual team on the console software team. The mythical Hydra name was already in use to describe our three-OS model, so after a bit of deliberation, we named ourselves after Hagrid’s three-headed dog in Harry Potter, and Team Fluffy was born. Over time, we expanded (and promoted) our way to eight Principal testers in console, and three in Live, for what I believe is the highest team concentration of Principal testers at the company. Someone asked once if we needed that many Principal ICs – but honestly, I don’t know how we would have made this product without the diversity and deep skills of this group. I think the eight of us all contributed significantly to getting this product shipped on time with quality.

Teams and Products

Yes – products are important, but I’m a huge believer in the team first – and I think it’s just as difficult and challenging to build a great team, as it is to build a great product – and that we should all put the same care and passion into hiring and developing our teams, as we do in building great software.

I’ll share a few other lessons later this week.

The Myth of the Myth of 10x

I first heard of “10x” software development through the writings of Steve McConnell. Code Complete remains one of my favorite books about writing good software (The Pragmatic Programmer, Writing Solid Code, and The Clean Coder are also on that list). Steve McConnell’s rarely updated blog (titled 10x Software Development, contains the following tagline:

Numerous studies have found 10:1 differences in productivity and quality among individuals and even among teams. This blog contains Steve McConnell’s thoughts about how to move toward the "10" side of that 10:1 ratio.

This article, and this one, are probably the best views from the McConnell side on 10x. They cover a lot of the studies and anecdotal information about the 10x concept. Although the original study cited a 10-fold difference in productivity and quality between different programmers with the same levels of experience, I use the 10x label to describe those software folks who “merely” differentiate themselves in terms of output and quality significantly more than their peers. While I will use the term “10x” throughout this post for conformance with the topic, it’s not an exact multiplication. For me, the term means, significantly higher output and quality.

The Myth?

Recently, there was a bit of buzz about this article about the Myth of the 10x Engineer. The meat of the article includes some valid criticism of the studies citing 10x Engineers, and as someone who is generally a skeptic, I acknowledge that the studies have holes. I think there are a lot of misconceptions about what a 10x engineer actually is, so the controversy doesn’t surprise me.

The article begins with the following statement:

I was a 10x engineer for 7 months and then I was a 0x engineer for a year and a half. You burn the candle at both ends. You end up with alcoholism and depression. You’re talking about a very small subset of people. And they might end up in divorce and financial ruin. When people think you’re a 10x engineer, they think you have skills that you don’t. You invariably let people down.

The full article (which, by the way, is very nicely written and worth the read) makes a very good case against the validity of the studies about 10x, and includes several statements from the author’s twitter followers supporting the quote above.

Not a Myth?

But I interpret 10x much differently, and think a lot of commenters miss the point. This, in my opinion, is at least partially due to the “10x” label.

But more importantly, 10x isn’t about working 10x as hard. It’s not even about working twice as hard. It’s not about effort. Being 10x (or significantly more impactful, or whatever you want to call it) is about making smart choices and taking care in your work. I’m sure if I were to put fifty similarly proficient developers in a room and ask them each to implement a known set of algorithms or implement well defined functionality into an existing app, or any other task where the problem and solution are largely understood, that there will be some variance in the time it takes each of them to finish the tasks, but the best in the group would maybe be twice as fast as the average – if that.

Where 10x software engineers excel is in adaptive challenges – where the problem may be understood, but the solution is an unknown (and where the problem may change as the solution emerges). 10x engineers recognize patterns, they know when they’ve hit a dead end, and they can zoom from the 10,000 foot big picture down to minutiae and back again in an instant. They aren’t significantly more productive than their peers because they work eighteen hours a day or even because they type at a whirlwind pace. They are more productive because they solve problems 10x faster than their average peer. They are more productive because there’s less waste in their work – they are careful enough to do things right the first time, but have enough breadth of knowledge that they find the correct choice sooner. You can’t make yourself into a 10x engineer by working harder or longer – you make yourself a 10x engineer by studying and practicing your craft, by investing in learning, and by challenging yourself to continuously make better decisions about how you work.

All that said, not everyone needs to be – or can be a 10x – and that’s ok. But I, for one, want (and love) to work with 10x developers so I can learn from them, learn how they learn, and perhaps discover a few tips to improve my own output and value.

Plus ça change..

In early January, I’ll hit my 19 year anniversary of working at Microsoft (I’ve worked for Microsoft 18.5 years, and as a vendor at Microsoft for the first 5 months or so). There’s been a lot of change over the years, but perhaps never as much as is going on right now.

We’re getting a new CEO, the review system has been paved, there’s a major re-org underway, and Xbox One ships in 6 days. I can honestly say that the two years I’ve spent on Xbox has been the most challenging, fun, and eye-opening experience of my career. I’ve learned a lot – I’ve had an impact – I’ve worked with incredibly smart people – I’ve made friends. I can’t rave enough about how great it’s been to be part of this team.

But it won’t be the same anymore. Sure, a lot of the people on the team will still be here, but you only launch a console once every 5-7 years. The Xbox software will continue to improve and games will get even better, but you only get to ship the console once. Of course, there’s so much more I want to do here, that there’s a reasonable chance I’ll continue to work on Xbox, but I can’t say for sure yet. I have other opportunities to weigh – and I’m in much need of a vacation and some rest (and some quality time with my family), and this is no time to make big decisions. I’ve managed to change my mind at least once a day for the past few weeks when thinking about what’s next…and just when I think I have it figured out, a new opportunity shows up.

I’m sure the rest and family time will lead me in the right direction.

In the meantime, I have a zillion things I’ve been meaning to write about, and no marathon debugging sessions to get in the way, so I expect to barf some random ideas onto the internet in the coming days and weeks. I may spend some more time on twitter (heck, I even posted to Facebook last night!).

I can’t wait to see what happens next.

Mobile Application Quality

I forgot to mention this before, but on Thursday, Jason Arbon from applause and uTest will be giving a practically free ($99) course on mobile application quality. Jason is a great teacher, and really knows his stuff in this area.

More information at the SASQAG web site (scroll down a bit on the home page)

Home Again

I implied at the end of my last post that I’d follow up after my keynote (I failed – sorry). This was a weird conference for me. While I attended nearly all of the keynotes, I only made it to a few other sessions, and didn’t have as much time to hang out with folks as I would have liked. For better or for worse, I spent the majority of the week in my hotel room working (internet access was much better than in the conference area). Even in hindsight, taking care of the day job was the right thing to do.

But I had a great time at STAR. I’m mostly happy with my keynote and think I delivered everything the way I wanted (incidentally, the keynote is online here). I thought Friday’s panel discussion with Jon Bach and Dawn Haynes was a lot of fun (although I probably couldn’t have been more annoyed with the content and style of Friday’s keynote speaker). Over all, it was another great STAR conference, and I can’t wait to attend another one.

And if you missed me, now that Xbox One is almost out the door, I’m planning to be back at STAR for STAR East this spring in Orlando. I hope to see some of you there.

One down, two to go…

it’s 6:30-ish pm, and I’m killing time waiting for my sound check for tomorrow’s keynote, and thought I’d do a quick brain dump of today’s tutorial session.

Today’s session was “Alan Page: On Testing” – which is a pretty wide open topic. For the slide handouts, I slapped together slides from a bunch of things I could talk about, but my plan all along was different. In a perhaps risky move, I decided that I’d take the first 10 minutes of the session to collect as many questions from the audience as I could, then I loosely grouped the questions and put together a few impromptu talks to cover the answers. I took questions as I went, plus a few ad-hoc questions at the end, and filled the 3.5 hour session.

The problem with this sort of thing is that it exhausts me. I’m wiped out, and I’ve lost half of my voice, but I should be good to go for my keynote in the morning.

The other drawback of this sort of session (and the few pieces of feedback I glanced at reflect this) is that this sort of session is polarizing, Attendees either got a ton of value, or little value – comments like “Love the unstructured format – tons of great information” were contrasted with, “Didn’t like the unstructured format – too much information”. I’m not too concerned, since the conference circuit isn’t really my thing, but I feel a little bad I didn’t set up the people in the “don’t like unstructured” group a little better with expectations.

More tomorrow after my (structured-ish) keynote.

Silos, Walls, Teams and TV

Between Netflix, Amazon Prime, and a busy schedule, I’ve gone without cable television for the last five or so years…up until a month or so ago when I bit the bullet (for a variety of reasons), and added a television package to our existing internet service. It’s working great so far, but the story about how we got everything working is worth sharing and pondering.

The Picture

On a Wednesday evening, I called Comcast to add a television package. I knew what I wanted, so ordering was easy. The sales person told me I need to schedule an installation, so we set something up for Sunday. There was no record of our house every having cable television before, but three years ago, I had service for a short time so I could watch the 2010 world cup. So – I had a hunch that I could take care of it myself, so I made arrangements to pick up a box and try it myself.

Thursday morning I grabbed a set top box from the retail store – where the sales person confirmed that I didn’t need an installation – just a “switch flipped” on the box at the end of our driveway.

On Friday night after work, I plugged in the cable box and watched to see what would happen. I half-way expected it not to work, and I was half-right. I was getting audio for all of the channels, but no picture. I was pretty sure things on my end were correct, so I gave support a call.

I’m pretty polar on my customer service experiences. When I have a good experience, I’m elated. When I have a poor experience, I’m annoyed at best. I called support, answered some questions, pushed some buttons, and eventually found a real person to talk to. I clearly explained my situation (new box, previous cable in the house, audio is loud and clear, no picture) to the support person. As I expected, they used a script, asking me if my box was plugged in and if the cables were connected (despite that I had already explained that I had audio and was connected via HDMI). After only a minute or two, I was able to get transferred to someone else. They tried rebooting my cable box a few times before putting me on hold to “talk to a colleague”. Soon after, I was disconnected, and when I called back, the support line was closed for the evening.

On Saturday morning, I called back, and went through the same process. This time, someone eventually figured out that “ I had the wrong account codes”. But – they couldn’t change them. They had to transfer me to someone in sales who could apply the correct account codes. The sales guys did their thing. They tried to sell me a bunch of stuff I didn’t want, but eventually “applied my codes” and sent me back to support – because I still had no picture.

I spent over 30 minutes with the next support person. I still had no picture, and she had me double check all of my connections, and insisted on rebooting my cable box a few times (remember folks, reboot-is-always-a-valid-solution), but no dice. Finally, I chimed in and said, “Hey – to remind you, I ended up here because my account codes or something like that weren’t applied properly – does that help?”. After an awkward silence plus about 3 seconds, a picture appeared on my television. Problem solved! She closed by asking (certainly a Comcast standard) if there was anything else she could do for me. As a matter of fact, there was. I asked her to verify the services I was signed up for and the price I was paying. Sure enough, the guy in sales screwed it up, and I was getting services I didn’t want for more than I planned on paying. So…she transferred me back to accounts.

I explained the situation to the person in accounts, but was told that the type of account I signed up for was an account outside of her department. Her department, apparently only handled the “standard” packages, and I needed to talk to someone in “specials” or something like that. She transferred me again, and within a minute or two, my service remained up and running, and I was getting the services I wanted for the price I planned to pay.

To be clear, the television service (on top of my internet service) has been flawless. But the path to get there was nothing but.


I could go back and count everyone I had to talk to, but in reality, the problem with my experience above is that I talked to way too many people. In fact, I’d say that I talked to too many people the moment I talked to the second person. In nearly all of the great support experiences I have experienced (and have read about), I’ve been taken care of my one person (sometime a few) who were empowered to help. At Comcast, employees apparently work in silos. They have a single activity they are trained in, and allowed to perform. I’m sure that makes sense on paper (to someone), but in practice, it’s a poor experience.

Making Software

Believe it or not, this post isn’t about my support complaints. It’s about making software – because I fear that many organizations are making software the same way that my cable company handles support. When you have a team of programmers who only program, testers who only test, analysts that only analyze, and managers who only manage, you have an organization that works slowly, and likely produces crap.

Not everyone, of course, has to be able to do everything, but everyone should feel empowered to do whatever is needed (within their capabilities). You should be able to write a spec, fix a bug, perform a test, or lead the team regardless of what your title says.

Generalizing Specialists

There’s a reasonable number of talks and articles amongst the agile community on Generalizing Specialists

By staffing a team with people who have an area of expertise, but can do anything, you can maximize the value of each delivery cycle.  (reference)

It’s worth noting that the article quoted above refers to Specializing Generalists – and although it appears that many folks use the terms interchangeably, I think there’s an important distinction.

A generalizing specialist is more than just a generalist. A generalist is a jack-of-all-trades but a master of none, whereas a generalizing specialist is a jack-of-all-trades and master of a few. (reference)

Again (because for some of you, I’ll have to say it again), I’m not saying that everyone has to do everything. Your “generalist” skills don’t need to span every software activity. But I am saying that you will be more valuable to your team (and your team will work better and make better software) if everyone in your organization can do more than specialize in just one thing. This is true regardless of the methodologies and approaches used on your team. Silos and walls create waste (wasted time, wasted efforts, wasted energy, etc.).

I really don’t understand why some software teams insist on having specific roles for specific team members and offer little to no encouragement to reach beyond their silos. Great results come from great teams – and great teams (whether it’s a sports team, a sales team, or a software team) work together and compliment each others skills.

See also: T-Shaped Personas (and E-Shaped)