HWTSAM – Six Years Later

We’re less than a week away from the sixth anniversary of How We Test Software at Microsoft (some chapters were completed nearly seven years ago).

Recently I was considering a new position at Microsoft, and one of my interviewers (a dev architect and also an author) said that he had been reading my book. I stated my concern that the book is horribly out of date, and that it doesn’t really reflect testing at Microsoft accurately anymore. In the ensuing discussion, he asked how I’d structure the book if I re-wrote it today. I gave him a quick brain dump – which in hindsight I believe is still accurate…and worth sharing here.


So – for those who may be curious, here’s the outline for the new (and likely never-to-be-written) edition of How We Test Software at Microsoft. For consistency with the first edition, I’m going to try to follow the original format  as much as possible.

Part 1 – About Microsoft

This section initially included chapters on Engineering at Microsoft, the role of the Tester / SDET, and Engineering Life cycles. The new edition would go into some of the changes Satya Nadella is driving across Microsoft – including flattening of organizational structures, our successes (and shortcomings) in moving to Agile methodologies across the company, and what it means to be a “Cloud First, Mobile First” company. I think I’d cover a little more of the HR side of engineering as well, including moving between teams, interviews, and career growth.

Part 1 would also dive deeply into the changes in test engineering and quality ownership, including how [combined] engineering teams work, and an intro to what testers do . This is important because it’s a big change (and one still in progress), and it sets the tone for some of the bigger changes described later in the book.

Although I don’t think I could give Marick’s Testing Quadrants the justice that Lisa Crispin and Janet Gregory (and many others) give the them, I think the quadrants (my variation presented on the left) give one view into how Microsoft teams can look at sharing quality when developers own the lions share of engineering quality (with developers generally owning the left side of the quadrant and testers generally owning the right side.

And finally, part 1 would talk about how community works within Microsoft (including quality communities, The Garage, and how we use email distribution lists and Yammer to share (or attempt to share) knowledge.

Part 2 – About Quality

Fans of the book may remember that the original name for the second  and third sections of the book were“About Testing”, and “About Test Tools and Systems” – but I think talking about quality – first from the developer perspective, and then from the quality engineer / tester perspective (along with how we use tools to make our jobs easier) is a better way to present the 2015 version of Testing at Microsoft.

I’d spend a few chapters showing examples of the developer workflow and typical tools – especially where the tools we use are available to readers of the book (e.g. unit testing and coverage tools available in Visual Studio and use of tools like SpecFlow for acceptance test development).

This section would also include some digging into analysis tools, test selection, test systems, and some of the other big pieces we use across all products and disciplines to keep the big engines running.

I didn’t talk much about how Microsoft does Exploratory Testing in the original book – I’d fix that in the second edition.

I also thought about breaking this section into “developer owned” and “testing owned” sections, but that line isn’t clear across the company (or even across organizations), so I think I’d start as a progression from unit testing to exploratory testing and let readers draw their own ideas of where their team may want to draw the line.

Part 3 – About Data Driven Quality

There’s at least a book or two of information that could go in this section, and it represents the biggest shift over the last six years. The original chapters on Customer Feedback Systems and Testing Software Plus Services hinted about how we approach Data-Driven Quality (DDQ), but our systems, and the way we use them have matured massively, and there’s definitely a lot of great information worth sharing – and enough where it’s an easy decision to dedicate a full section and several chapters to the topic.

Part 4 – About the Future

Much of what I talked about in this section of the original has either happened…or become obsolete. As in the first version, I would reserve this section for talking about engineering ideas still in the experimental stage, or ideas currently under adoption. Also, as in the original, I’d use this section as a platform for whatever essay idea chewing on me about where software engineering is going, and what it means to Microsoft.

Part 5 – Appendix

In my 20+ years of testing, I’ve read at least a hundred different books on software testing, software development, leadership, and organizational change. I’ll include a list of books that have had a significant influence on my approach and views to software engineering, publications that influenced the content of book, and publications that readers may find helpful for further information.


  1. It’s been very interesting to have read your book in the formative days of my testing career and to have seen the changes play out at different jobs that I’ve had. This book formed my ur-ideas about what testing could be.

    This book convinced me that I wanted to follow the path of an SDET. When I eventually got that job, I decided it’s more important for developers to have ownership over the tests. I don’t really know what huge numbers of automated tests mean if developers don’t have an investment in them or care about them failing. It didn’t shock me when I heard that Google was getting rid of the SDET role. I’ve also heard you complain about automated GUI testing many times, as this is how the SDET role is often envisioned. I guess you replace that with data and monitoring? (Hint: there is space for you talk about coverage without these types of tests).

    Just as you are writing about there being big changes in the relationship of these roles in “combined” teams and Elisabeth Hendrickson is experimenting with this at Pivotal Labs on Cloud Foundry, I’m working on my own experiments of how this changes the equation if you work at a smaller place. My guess is that I’m not the only solo tester at a small company working on this.

    One result of this model that we don’t really talk about enough in testing or software is the friction of having people collaborate across roles while still having strict job titles and the management chains that support them. This is what I, and I suspect others, have to work through on a daily basis: getting the collaboration to happen while we balance the hairball of red performance tape and the managers invested in keeping the red tape going.

    Thanks for continuing to write and blog about getting beyond that hairball. I’ve used your writing more than a few times when I’m arguing for more collaboration. It makes things easier when I can send out a link and say, “this is what Alan is doing,” and the article is a clear illustration of what’s happening and why the changes have been made.

    HWTSMS was my introduction to your voice in the testing world and that’s something I’ve found timeless and
    irreplaceable. My signed copy shall stay exactly where it is on my bookshelf, and I hope you keep writing and blogging.


  2. I read HWTSAM after I accepted a position at Microsoft, but before my start date (Jan 2009) as a way to get to know the lay of the software-testing landscape at Microsoft. Considering the variety between team at Microsoft, and the rate of change, HWTSAM was an audacious attempt…. and a successful one. Even over the years as I have perused it, it gives perspective and history to how and why things are the way they are today at Microsoft.

    One a slightly tangent topic, since Alan mentions DDQ, any readers looking for more info on it may want to check out:

    Steve Rowe’s series of posts on DDQ: http://blogs.msdn.com/b/steverowe/archive/2014/06/16/data-driven-quality.aspx

    Ken Johnston (HWTSAM co-author) explains EaaSy and MVQ: http://blogs.msdn.com/b/kenj/archive/2014/05/20/the-future-of-quality-is-easy-with-eaasy-and-mvq.aspx

    *I* introduce you the questions of DDQ: http://www.setheliot.com/blog/2014/11/24/ddq1/ (also see Alan’s / Marick’s Testing Quadrants above)

  3. I found out in October 2014 that you had quoted me on page 292 of HWTSAM, a wonderful thing to learn 7 years into my retirement! Thank you!

    I now try to maintain as small a digital footprint as necessary so it has taken a while for the Brownian-motion of random communications to filter my way.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: