HWTSAM – Six Years Later

We’re less than a week away from the sixth anniversary of How We Test Software at Microsoft (some chapters were completed nearly seven years ago).

Recently I was considering a new position at Microsoft, and one of my interviewers (a dev architect and also an author) said that he had been reading my book. I stated my concern that the book is horribly out of date, and that it doesn’t really reflect testing at Microsoft accurately anymore. In the ensuing discussion, he asked how I’d structure the book if I re-wrote it today. I gave him a quick brain dump – which in hindsight I believe is still accurate…and worth sharing here.

So – for those who may be curious, here’s the outline for the new (and likely never-to-be-written) edition of How We Test Software at Microsoft. For consistency with the first edition, I’m going to try to follow the original format  as much as possible.

Part 1 – About Microsoft

This section initially included chapters on Engineering at Microsoft, the role of the Tester / SDET, and Engineering Life cycles. The new edition would go into some of the changes Satya Nadella is driving across Microsoft – including flattening of organizational structures, our successes (and shortcomings) in moving to Agile methodologies across the company, and what it means to be a “Cloud First, Mobile First” company. I think I’d cover a little more of the HR side of engineering as well, including moving between teams, interviews, and career growth.

Part 1 would also dive deeply into the changes in test engineering and quality ownership, including how [combined] engineering teams work, and an intro to what testers do . This is important because it’s a big change (and one still in progress), and it sets the tone for some of the bigger changes described later in the book.

Although I don’t think I could give Marick’s Testing Quadrants the justice that Lisa Crispin and Janet Gregory (and many others) give the them, I think the quadrants (my variation presented on the left) give one view into how Microsoft teams can look at sharing quality when developers own the lions share of engineering quality (with developers generally owning the left side of the quadrant and testers generally owning the right side.

And finally, part 1 would talk about how community works within Microsoft (including quality communities, The Garage, and how we use email distribution lists and Yammer to share (or attempt to share) knowledge.

Part 2 – About Quality

Fans of the book may remember that the original name for the second  and third sections of the book were“About Testing”, and “About Test Tools and Systems” – but I think talking about quality – first from the developer perspective, and then from the quality engineer / tester perspective (along with how we use tools to make our jobs easier) is a better way to present the 2015 version of Testing at Microsoft.

I’d spend a few chapters showing examples of the developer workflow and typical tools – especially where the tools we use are available to readers of the book (e.g. unit testing and coverage tools available in Visual Studio and use of tools like SpecFlow for acceptance test development).

This section would also include some digging into analysis tools, test selection, test systems, and some of the other big pieces we use across all products and disciplines to keep the big engines running.

I didn’t talk much about how Microsoft does Exploratory Testing in the original book – I’d fix that in the second edition.

I also thought about breaking this section into “developer owned” and “testing owned” sections, but that line isn’t clear across the company (or even across organizations), so I think I’d start as a progression from unit testing to exploratory testing and let readers draw their own ideas of where their team may want to draw the line.

Part 3 – About Data Driven Quality

There’s at least a book or two of information that could go in this section, and it represents the biggest shift over the last six years. The original chapters on Customer Feedback Systems and Testing Software Plus Services hinted about how we approach Data-Driven Quality (DDQ), but our systems, and the way we use them have matured massively, and there’s definitely a lot of great information worth sharing – and enough where it’s an easy decision to dedicate a full section and several chapters to the topic.

Part 4 – About the Future

Much of what I talked about in this section of the original has either happened…or become obsolete. As in the first version, I would reserve this section for talking about engineering ideas still in the experimental stage, or ideas currently under adoption. Also, as in the original, I’d use this section as a platform for whatever essay idea chewing on me about where software engineering is going, and what it means to Microsoft.

Part 5 – Appendix

In my 20+ years of testing, I’ve read at least a hundred different books on software testing, software development, leadership, and organizational change. I’ll include a list of books that have had a significant influence on my approach and views to software engineering, publications that influenced the content of book, and publications that readers may find helpful for further information.