Time Flies

I wrote a bit of a career retrospective just a few months before I joined the Communicator team at Microsoft (the series is captured in these posts: (part 1 is here, part 2 is here, part 3 is here, part 4 is here, and part 5 is here).

June 6, 1995 – exactly 15 years ago today – was my very first day at Microsoft. No need to retrospect again, but wow – what a variety of experiences I’ve had. I never thought I’d work at Microsoft this long, but I have no regrets.

Here are some non-traditional stats I thought I’d share.

  • Number of offices:15
  • Number of buildings I’ve worked in: 7 (Buildings 4, 10, 27, 118, 21, “Westpark”, and 30)
  • Most number of offices in one building: 5 (building 27)
  • Number of managers: 12 (including one manager for nearly 6 years)
    • Note: two of those managers lasted only a week
  • Number of projects I’ve worked on that were cancelled by BillG: 1 (in 1999 or so, I was part of a small virtual team working on some digitial tv stuff)
  • Number of programming and script languages I’ve learned 9 (in mostly chronological order: Visual Test, C, C++, Perl, VBScript, x86 Assembly, C#, Python (I’ve used all of these for testing except Python)
  • Number of shipping products I’ve worked on: 11 (not bad, considering that I spent 5 years in EE not working on products)


  • Number of employees at MS when I started: ~7500 (compared with ~90k today)
  • Number of those 7500 still at MS: ~4500
  • Number of those 4500 with the same start date as mine: 30

There’s probably more, but that’s all that comes to the top of my mind. Not bad for only 15 years.

It must be simple

Many of you know that I’m a big fan of the five orders of ignorance (link is to my interpretation – the author’s original paper is here). I especially like the concept of “unknown unknowns” those bits of knowledge that are critical – despite the fact that we didn’t recognize those bits until they fell onto our laps.

I’ve been pondering over an interesting fuzzy line between Armour’s second level of ignorance (you know you don’t know something), and his third level of ignorance (you don’t know you don’t know). I’ve noticed that often, when you know nothing about a subject, you perceive it as simple. Then, as you learn more, it seems impossibly difficult; and then, as you master it, it become’s easier to manage (but probably never as simple as you imagined in the first place). The second and third levels of ignorance seem to play with each other as you try to move knowledge into the first level of ignorance (you know what you know)

An example would probably help here. Many years ago, I knew nothing about recording music, but I wanted to be able to record my own stuff, overdub, mix, edit, etc. Since I had never done it before, I was convinced it was easy, and that anyone with a brain and a halfway working set of ears should be able to do it.Within a few weeks of attempting to pick up this “easy task”, I realized it was extremely complicated and required a lot of meticulous work (this was pre-digital, so within a few weeks I was fixing timing issues with a razor blade). I didn’t think I’d ever get a handle on it, but eventually I figured enough stuff out for my own purposes, and to this day I still uncover areas of complexity I never imagined.

There’s another great example of this from my list of favorite movies that came out when I was 19 – Better off Dead. There’s a scene where John Cusack’s character is on top of a mountain ready to ski down and his friend gives him this advice:

Go that way, really fast. If something gets in your way, turn

That’s exactly the “expert” advice someone who doesn’t know the subject would give. Steve Martin used to tell a similar joke about skiing – I don’t remember it exactly, but it went something like this:

Skiing – you go to the top of a mountain with slippery things on each foot and go down. Try not to – that would be a sport!

Perhaps a better example for testers is Jerry Weinberg’s Perfect Software book. To me, it’s not so much a testing book, but it’s a wonderful book to give to people who don’t know anything about testing – mainly because most people who don’t know about testing often think it’s a simple activity that has something to do with banging on keys or pushing buttons. Once you learn a bit about software testing, you of course realize that it’s far more complex than you ever could imagine. Eventually (if you’re lucky), you fall in love with the challenge and never look back.

I see at least a few examples of this phenomenon every day. I have lost track of the times I’ve heard people critiquing the efforts to plug the oil leak in the Gulf and offering their own advice. However, the fact that they haven’t been able to solve it, and have tried to fix it several ways must mean it’s probably a more complex problem than most of us understand (or that the efforts are run by aliens from the planet Stupid).

I suppose the moral of all this is that nothing is as simple as it appears – especially with knowledge acquisition. Keep it in mind the next time you think you know something …

Are you a game changer?

I took an internal training course last week where we heard lectures from a bunch of MS execs (I don’t do it justice – it was far more interesting than I anticipated and something I’d do again in a heartbeat).

Anyway, one of the lecturers was talking about software markets and showed a diagram that looked something like this:

Picture1The basic concept was that there’s a base set of functionality or features that a product has to have to compete, as well as areas where it needs to be better than (or at least as good as) competitors. “Hot” products also have something unique – a “game changer” that helps drive adoption.

With the iPhone for example, the foundation was that it had to make phone calls and play music.The touch controls, the camera and photo viewing and some of the other similar features made the iPhone competitive.

The apps, the appstore, and the “there’s an app for that” wildness is huge – I know dozens of people who use an iPhone only for the apps. That is a game changer.


Of course, this is a lot like Kano – same idea, different names.

imageKano has “must-haves” – the features you, well – must have. The “one-dimensional” properties in Kano are kind of like the differentiators in the chart above, and the “attractive” needs in Kano are sort of the game changers.

The thing I like about Kano is the way you plot the data. For each feature you’re considering, you ask customers (a focus group works great for this) the functional and dysfunctional form of their thoughts. For example, “How would you think of shimmery blue menus”, and “How would you feel if we didn’t have shimmery blue menus”. Their responses help you plot out where on the chart the particular feature lives.

Then it’s up to the decision makers to use the data and come up with the right mix of features that will satisfy customer needs – and give them enough “oomph” that they’ll want to give you money.


That’s all sort of interesting (or not), but I sat down here to write about something else. I’ve known about Kano for years, but when I saw the triangle version last week I had an insight.

The triangle also describes your career

If you want to be successful, at the very least you have to be able to do your job – that’s the foundation. Lot’s of people do just this and get by just fine. The next level (differentiation) is where you get good at your job – possibly even excel. There are a lot of people here too – these are your typical “great” performers. In many companies, these are the cream of the crop – and they should be.

Then, you have the game changers. There aren’t many of these, but they kick ass. They always have …something – something that only they bring to the table. These are the people that innovate and come up with the ideas that the differentiators use to get ahead (building, of course, on their execution and foundation of knowledge). Many people want to be game changers – some even proclaim they are game changers, but this level is reserved for people who earn it through great ideas and insanely hard work.

So how about you – do you want to be a game changer?

Network and influence

I gave a short talk to an internal MS community this week. The topic of the day was “influence”, and I thought it was appropriate to talk about the value of building (and maintaining!) an informal network – and the impact of that network on influence. The advice to have an informal network isn’t new – I bet it’s mentioned in the majority of books on leadership, but I don’t think enough people who want to be leaders (or be influential) take the point seriously enough. Your network is a resource for discovering new information as much sounding board,for ideas you want to share.

Your network is also an opportunity for you to build credibility. I know people who claim to be leaders. Sometimes they claim to be a leader because they’re in a position of authority. Other times, their “leadership” is just a self proclaimed act. In my world, you aren’t a leader until other people say you’re a leader. You don’t get it any other way than building credibility and trust among those you want to lead – and your network is a great place to build that credibility and trust across people who don’t have to listen to you if they don’t want to.

There are other values of a network. In HWTSAM, I talked about the Test Architect Group (TAG) at Microsoft. TAG is a collection of senior testers who meet regularly to talk about a variety of testing topics.

The value of having Microsoft’s most senior testers regularly review, brainstorm, and dissect solutions for complex test problems is immeasurable. In recent years, TAG has become something of a sounding board for new thinking, new methods, or new tools in testing. Presentations and demonstrations of ideas and implementations from test groups spanning every Microsoft division fill many of the meeting agendas. The value and depth of the feedback that the TAG provides is respected and sought after. A few meetings a year are reserved for “TAG business,” which includes discussions about company-wide initiatives driven by TAG
and other projects where TAG is a significant contributor (such as the MSDN Tester Center).

Perhaps the largest benefit of the regular meetings is in the value of networking. The extensive peer discussions and the view into the variety of work done across the company that is presented give TAG members much of the knowledge and information they need to make strategic decisions that affect the entire company.

I’ll say it again because it’s huge – the value of this particular tester community is the network. The fact that we talk to each other about what we’re working on gives us a small-world network – a huge reduction in degrees of separation between just about any testing knowledge in the company.

Yet – many people don’t take advantage of their opportunities to build their network. They don’t have the time – they think their day job is more important, or have something else to do they think is more important. It’s hard – last week I was in a great “hallway conversation”, but I found my mind wandering as I wondered if I should really be back in my office doing “real work” (I didn’t, now I’m behind, but it was worth it). Alas, some people who do consistently prioritize their commitment to building their network find that their superiors don’t share the priority and look down on work spent “outside of their core job” (my advice for these people is to find a new job).

I was asked once whether it was leaders who built great networks, or if people who built great networks became leaders. My old answer was that it was a toss up. Today, as I think about this more, I realize that there are a great number of leaders who got there because of their effort to build, maintain, and nurture a huge network of informal relationships. I’ve also seen leaders (including exec level leaders) fail because they thought that their position was enough, and didn’t work to build their network, credibility, and trust.


Don’t forget the checklist

I recently read The Checklist Manifesto: How To Get Things Right by Atul Gawande. This is yet another book not about testing that testers should read. Gawande is a surgeon and relates his experiences from the field of medicine throughout the book. )ne of the first chapters goes into depth on the growing complexity and body of knowledge in medicine, and how difficult it is to keep many of the important details of the practice in mind consistently. Sound Familiar?

In Gawande’s experiments, he looked at areas where there were “bugs” (“bugs” in his case were complications or death), and created short, simple checklists that could be used by the medical staff to reduce risk – and the checklists worked wonderfully – fore example, reducing infection in central line procedures by over 60%. He also points out the cause of error (I’m a huge fan of Reason’s Human Error) usually breaks down to errors of ignorance (not knowing the right thing to do), and errors of ineptitude (not knowing how to apply what we know).

Checklists aren’t new to testing. Kaner talks about checklists, and testers at Microsoft have been using “am I done” type checklists for at least 15 years (Michael Hunter has a great list here). Checklists are good tools for testers to use and probably worth looking at more closely.

But building a checklist is probably harder than you think. In my experience (Gawande saw the same thing), most checklists are too long and contain too much detail. They get in the way! To make matters worse, as you learn more, you will likely add additional items to your checklist. The solution is simple enough – start with the smallest possible checklist you think you can get away with (and then see if there’s anything else you can remove). Realize that your checklist can change over time as you learn more about your processes and the types of errors your team makes. Try it – I think you’ll find that checklists are a great help.


Remember -  don’t forget to use a checklist (you can always use another checklist as a reminder)

Peering into the white box

In a conversation last week at STAR, someone was talking about how far testers should debug an issue. Personally, I don’t like it when testers only report the surface of the bug, and I think good testers always dig a little deeper to see what other more interesting issues may be lurking a layer or two below. It can certainly be a bit of a time sink, however, if testers investigate every single bug to the line of code causing the problem. The interesting comment I heard in this particular conversation was the statement that “testers should never look at the code”. The points I heard in defense were all in line with the cons of white box testing – e.g. “testers need to focus on the customer experience, and looking at code ‘taints’ this focus”, or “it moves tester focus to non-essential testing”, etc. I, for the most part, agree, but I think there’s a hidden (but real) value in applying the tester mind to looking at product source code.

In my experience, testers are excellent code reviewers, and think testers should be involved in code reviews much more often than they are today. If you think about it for a moment, it makes a lot of sense. It seems that every week, I read another article or blog post explaining why developers aren’t good testers (“they only focus on the happy path”, “they don’t think of negative conditions”, etc.). There are also numerous articles and experience reports about the value of code reviews for finding bugs earlier.

Now, if you put 2 and 2 together, testers reviewing code must be a great way to find bugs early. When developers review code (and I’ve watched them), they look for the basic general programming errors, but mostly verify that the code does what it’s supposed to do. Testers, on the other hand, question the code – they ask “under what circumstance would this line of code fail?”, or what-if questions like “what if the input values were unexpectedly large?”. Testers also think about how they would test the code – simply asking “how would I test this” often unveils more problems.

I imagine that for some testers, finding bugs in code isn’t as fun as spotting real product failures, but it sure seems to me that some testers reviewing some portion of the product code is a pretty darn good idea.

I’ll tell you how to get started next time.

Stop Guessing about my STAR presentation

For anyone who recently attended STAR East and saw my presentation, thank you for attending, and for reading my blog. Some of you have viewed the slides on the proceedings CD and noticed that it’s not really like what I presented. I tend to change quite a bit of material between the date the slides are due and the actual presentation (In fact, I added a new slide about an hour before the conference).

Here are the slides I presented last week. Stop Guessing How Customers Use Your Software

And here is the video I showed about what happens with Windows error reports.

Twinkle Twinkle I’m back from STAR

It’s Thursday night, and I have just returned home from STAR East. Lee Copeland said that he thought it was the best STAR conference ever, and I know many who agreed with him. As for me, I had a great time, but was a bit too distracted from trying not to suck at my keynote that I couldn’t take it in as well as I hoped. I give a lot of presentations, and can’t remember the last time I was nervous before a presentation – but I was a bit nervous before my Wednesday afternoon keynote. I thought I had reasonable material, and I’m usually pretty good at keeping thoughts straight in my head, but I would be talking to a room filled with peers who I really respect. I’ve given keynotes before (not at STAR), and presented to large groups before, but if I messed up (or worse, if I was boring), I knew that there were people in the audience who would talk about me on the internet! I once gave a keynote a few years ago and I had the distinction of following Joel Spolsky (I’m fairly certain he was a lot better than me, but at least nobody tweeted aboutit).On top of that, Elisabeth Hendrickson gave what was probably the best presentation I’d ever seen on Wednesday morning in her keynote session, so I was feeling sort of intimidated,So, I tweaked a little more, and went through my slides a bit more than I usually do. Then I paced. I never pace… – but I did more laps than I’d like to admit around that ballroom while I got my act together.

Then, for the next fifty minutes, I told some stories and shared a few ideas with several hundred of my best friends in testing.

And no one booed!

But it’s better than that. Some of those same respected colleagues ( those responsible for my nervousness) told me that they liked the presentation – that was very nice to hear, and each of those comments were probably the highlight of my week.

But there were other highpoints from the week – I met at least half a dozen people that I knew from twitter and blogs, but had never met in person before – each and every one of these meetings was a memorable moment for me and more good memories. Joey McAllister and I had dinner Wednesday and talked about music (a part of my life that I don’t get to talk about enough), and I got to talk to Lanette and Matt a bit more about details of my recent job change. I signed a few copies of hwtsam, but also signed Beautiful Testing for the first time (and second, third, and fourth times too).

And now it’s back to reality of my day job. For the first time I can remember, I have no conference plans on my calendar (or any business travel for that matter). I finished a chapter for Dot Graham’s upcoming compilation last week, and have no other (external) writing projects lined up (except this blog). This is good, because I have a lot of really cool work on my plate, a great team and a pretty awesome product to ship. Fun stuff.


For some reason, I have a lot of friends from the UK (could be soccer, could be Microsoft – I don’t know). I hang around them enough to know not to look down when someone says “Pants”, and that “sixes and sevens” have nothing to do with math. Learning the subtleties took time, and I misinterpreted conversations often enough to be embarrassed (or at least confused). We speak the same (roughly) language, but important meaning is sometimes lost.

Back in my day job, I asked four testers I know what terms they would use to describe some terms for the following scenario:

Say you have a simple test (or check) that verifies that the xyz driver is loaded when a USB XYZ device is plugged into the system. And – you want to run this test on 32bit and 64bit Windows, and with at least four different USB firmware implementations.

So – you have something you want to verify – let’s call it an “idea”, and you want to confirm your idea on some different configurations – seems pretty straightforward.

  1. What do you call the “idea” (plug in the device and make sure the driver loads)? Is that a test case?
  2. What do you call the permutations? Do they have a different name or are they just more of #1 – i.e. do you still have one test case, or do you have eight?

I have my own ideas (and I wrote about them in hwtsam), but when I asked four testers this question, I came up with five answers (adding mine to the mix).

I worry (perhaps too much) that lack of common vocabulary causes problems in testing. My colleagues who are in favor of certification (or at least aren’t violently opposed to certification) say that the common vocabulary encouraged in the various Bodies of Knowledge are the best things about the certification. But – since there are a variety of certifications (and a larger variety of interpretations), I can’t say that I expect certification will solve the vocabulary problem. But I can’t help but wish that we could dissolve the language gap.

Then again, perhaps it’s not that big of a problem. We do find ways to talk, and we do make (some) progress. For example, I was in a twitter conversation last week (or as much of a conversation as you can have in 140 characters) – I said “The problem with blah is that x, y, and sometimes z” – and the response was “In my world, I call that funklestein”. OK – great – that’s not my word, but now we can have a conversation, and it wasn’t that hard to get us there. I wish all testing conversations were that easy.


I’m wrapping up a ton of stuff this week in order to prepare for being away at star east next week. I hope to get a chance to meet some of you there. Come find me.