HWTSAM–Five Years Later

Five years ago this week, How We Test Software at Microsoft hit the shelves. The book has done well – both in sales, and in telling the story of how Microsoft approaches software testing.

An unfortunate aspect of writing a book like this is that after five years, most of the book is obsolete. Sure, there are some nuggets that stand the test of time, but so much has changed – and is changing, that it’s hard for me to recommend much of it as practicum. I don’t know if a new version is warranted – or necessary, but you never know what will happen.

To be clear, I’m still proud of the book, and think it has a lot of great ideas that will help many companies – it’s just not an accurate reflection of how Microsoft tests software. Here’s a section by section recap / overview of what’s changed….

The New World

Section 1 – About the Company

Now that we’re a Devices and Services company, the description of our product groups is flat out wrong. Beyond that, our organizations are much different – our test dev ratios are changing (from nearly 1:1 at the time of writing to nearly 2:1 dev:test today).

Five years ago, a few teams used adaptive (agile) engineering approaches, whereas now, nearly every team in the company is using or experimenting with some flavor of adaptive development. We ship almost everything faster. Testers are moving to analysis of end-to-end scenarios and data analysis, while “traditional” functional testing is moving (where it belongs), to the programmers. The career path examples we showed in the book have all changed – and will likely change again as the company continues to adapt and grow.

Section 2 – About Testing

The techniques described in the about testing section are all still relevant (although these days, I could easily increase the chapter on test design by about 50 pages and keep it full of good ideas). What’s changed most (and what will continue to change) is how much of the activities in this section are carried out by developers (and how much more of will be carried out by the developers in the coming years.

I’m still particularly happy with the chapter on Model-Based Testing (and thanks to Michael Corning and Harry Robinson for their help with the ideas for this chapter).

Section 3 – Test Tools and Systems

The section on bug reports is good – as long as you have the need to track bugs formally. Many teams can (and do) get by with fixing bugs as they find them – with no need to document the bug. I’ve also become even less of a fan of tracking test cases, but the examples are probably still valuable for those who need to do this.

It’s hard to believe that five years ago, I failed to recognize the critical relationship between test automation and test design (they’re in different chapters!). Although the technical information is correct, I would like to take The A Word and shove it into chapter 10 as an addendum.

The customer feedback systems I wrote about have expanded massively – and are much more of a focal point these days – both in the data we gather, and how we use it.

And finally (for this section), what blows me away is how much different our approach is to testing web services. I don’t think Ken will be mad if reveal that this chapter is embarrassingly obsolete. This isn’t because what we were doing then was bad…it’s just that we’ve grown so much in our approaches here that our five-year old processes seem obsolete in comparison.

Section 4 – About the Future

I suppose this section is still relevant  – and most of the future as I saw it in 2008 is reality today (although much of this section, in hindsight, could have been better written.

Next?

In a perfect world, I’d write another HWTSAM. Maybe someday, I will, but it’s not on the roadmap for today. There are too many big changes coming (both planned by MS, and spinning in my head) for me to put a stake in the ground. Or maybe, software is changing so much that trying to capture how a company does it at a point in time is impossible. I’ll need to think about it.

For those of you who bought (and read) HWTSAM – thank you. There’s still some good stuff there, but read it with a grain of salt (or at least from a lens knowledgeable that it’s a half-decade old book).