Last week I came across this article about Samsung fixing an SMS bug in their software. For the most part, it’s the typical “we found a bug and we’re fixing it” story that are just a bit too common in the news these days – but I was struck by this line:

Another annoying bug is related to the fact that the device cannot maintain phone calls longer than 3 minutes without rebooting.

To me, that’s a much more interesting (and critical) bug than an SMS issue. Yes – I do realize that a lot of people use their phone far more for SMS than making a phone call, but come on…

But I’ve also been wondering lately about tester judgment (there’s probably a better word for this, but for today, it will do). How do testers determine if a bug is a bug anyone would care about vs. a bug that directly impacts quality (or the customers perception of quality)? (or something in between?) Of course, testers should report anything that may annoy a user, but learning to differentiate between an “it could be better” bug and a “oh-my-gosh-fix-this” bug is a skill that some testers seem to learn slowly.

I once saw someone post a bug on their blog and ask readers to identify it. It was interesting to see several replies pointing out minor issues that may or may not have been bugs before someone finally(?) saw the critical bug. This wasn’t an isolated issue – I sometimes see testers so anxious to find a bug that they force the issue, trying to make a bug appear where there isn’t one, that they miss a bigger issue right in front of them. Other testers, however, seem to always be able to hone in on the biggest, most relevant issues.

So what is it that makes some testers zero in on critical issues, while others get lost in the weeds? Domain knowledge is a small part of it, but certainly not a critical factor in my observations. Critical thinking is a bigger part of it – as is experience, but I haven’t yet figured out to help testers consistently get out of the weeds.

Certainly not a huge issue, but if you have thoughts, please let me know.

I’m sure by now that most people who want to have seen the webinar I gave last week on test design (if not,the recording is here and slides are here).

I had a few goals in this presentation. One I stated clearly – this presentation was primarily about the how of Test Design – not the what. This means that I didn’t share a big list of test design ideas – in fact, I don’t think I shared any unless they helped make a point.

Other points I left (purposely) up to the imagination. I have an inkling that some testers think that there’s a master list of test design ideas somewhere, and once they have the list, they’ll be up for any testing challenge.

This, of course, is a horribly broken idea.

First of all, Test Design ideas (I’ve explored, and continue to explore the pattern metaphor with these ideas) are endless. Like infinity, as many testing ideas as you can list, I can think of one more.

The bigger challenge is knowing when to use which testing idea – and to me, that’s the bigger challenge in test design. Let’s say that I’m an expert in Boundary Value Analysis. You can stop laughing now – all that means is that I’m a hotshot at finding boundary issues. The problem is that only a small (but significant) percentage of testing actually deals with boundaries. I may try testing boundaries in a lot of places, but most of the time that effort isn’t doing anything for me.

As a tester, you need to recognize the problem your facing, then pick the best test design ideas for that problem.

Here’s a real example. It’s no secret that I like model-based testing. Sometimes, testers will ask me “how would I use model-based testing in this scenario?”. That’s a nice question, but it’s the wrong question. To be a good test designer, you need to look at the problem first (I need to test this application), analyze how the program works, then pick the best approaches. If you start with the approach and try to make it work, you’re probably going to fail. With this example, a tester with MBT in their “tester toolbox” may recognize a portion of the application that 1)behaves like a finite state machine, and 2) recognizes risk in traversing that state machine. Then they apply the technique.

The point is that to be a good test designer (and tester), you need a lot of testing ideas, and you need to know how and when to apply them.

And that’s really, really hard – and as you know, that’s also why it’s really, really fun.

 

Happy Testing

Some combination of fever and remnants of generic brand Sudafed from the building 30 first-aid cabinet seem to be forcing me to perform another round of keyboard pounding hoping that something close to comprehensible English will spew forth.

We’ll see how it goes – luckily this is more or less a personal blog (where I happen to talk about testing 95% of the time), so I don’t have to worry so much about losing clients – just about losing readers.

I observed a few things on my flight here that are so universal that they probably don’t need to be shared – but I thought I’d go ahead anyway.

• Every flight has someone with a "personal item" that’s as big as the suitcase I used when I moved to Wisconsin (another story). That person also boards near the end of the line and is surprised when they can’t find a place to put their "bag"
• If you are sitting in an aisle seat (which I always do), the person on the inside of you will get up to use the restroom at least 2-3x more often than you do
• Conversely, if you are on an inside (or window) seat, the person next to you will doze into an unwakeable slumber just as the seatbelt light goes off, and will not stir until the plane lands
• The person in the aisle seat who goes into an unwakeable slumber is nearly always sitting directly in front of me. They manage to put their seat all the way back just before they doze off, leaving the back of their seat six inches from my nose. If I’m lucky, their greasy hair flops over the back of the seat just enough that a few hairs fall into my food
• I’ll probably never understand the urge to jump from your seat and pack into the aisle the moment the plane pulls up to the gate. I’ve noted that the ferocity of people to pack into the aisle increases dramatically if I have a connecting flight departing in thirty minutes or less
• The distance a backpack extends away from the back of the wearer is directly related to the non-awareness of the backpack bulk from the wearer. In other words, pay attention you stupid f&*# before you whip your body around to grab your man-purse and smack me in the face with your extendo-pack

I will, of course, watch for deviations and more proofs of these observations on my return trip.

The hotel I’m staying at in Munich is quite nice. The bed is comfortable, the room is quiet, and breakfast is provided. As a vegetarian, hotel breakfasts are often a bit of a challenge. As I expected, sausages and bacon were in full supply. I do eat eggs (occasionally), so I’ve been grabbing a small spoonful of those and some of the slim pickings of fruit – and bread. The bread’s really good, so I’m thankful, but I may try to improvise something more interesting for myself tomorrow (I’m thinking table-made egg-white salad with some of the good mustard kept near the sausages).

I like to workout when I travel. Actually, I like to work out all of the time, I just tend to have that precious extra hour in a day when I travel. Before it sounds like I’m complaining, I’d like to say again what a nice hotel it is – at least as far as my room goes… So Sunday morning I was up about 6:00 and decided to head down to the gym on the first floor. I followed the signs and was sure I was lost. I was heading toward a dead-end and saw what looked like real hotel rooms..but just past the last one was a door that said "fitness center" – so I put in my card and walked in.

Sure enough, when I checked the fire escape map on the back of the door, I saw that the room next to the fitness center was a real room. I’m going to hope that they only use that room when the hotel is packed – but who knows (apologies if you were trying to sleep during my adventure).

The "fitness center" is about half the size of my hotel room and contains a treadmill, an elliptical trainer, an exercise bike, and a multi-purpose weight bench. My exercise drug of choice is running, so I hopped on the treadmill and started pushing buttons until it began moving. I was just placing an ear bud in my ear when I felt like I slipped on a piece of ice. "Hmmm..", I thought – "the belt has a bit of a slip in it". I’ve run on slipping treadmills before, and while a bit awkward, I can usually get by. I stepped up the pace a bit more and began converting Kph to Mph in my head when it slipped again. This time it was like someone grabbed my foot with a soft hand and pulled it backwards at warp drive. I managed to hit the stop button and keep my chin from smacking into the console, and decided that perhaps the elliptical machine was a better choice for me.

I hopped on the elliptical and began…ellipticizing (is there a better name for it). I was just getting in a groove when I heard the screech of a mouse. I quickly realized it wasn’t actually a mouse – it was just a huge squeak from the elliptical. I turned up the volume and kept going.

But I couldn’t drown out the screeches of the mouse under my feet. I kept turning up the volume, but the squeaking wouldn’t stop. Finally, when I was sure that my eardrums were about to burst, I gave up and moved to the bike.

Except the bike was broken – so I vowed to walk a mile or two extra in my exploration of Munich and headed back to my room.

As I’ve mentioned recently (and probably way too many times by now), I’m in Munich to give two presentations at OOP 2011. My first presentation is tomorrow. Tomorrow evening, in fact, in the “Night School” track. Tomorrow’s schedule has tutorials in the morning, lunch, two keynote sessions, and then, after a short break, 4 concurrent sessions running from 6:45 to 8:15. I’m a little worried about attendance – I think this conference is pretty big, but after tutorials and two keynotes, I’m worried people won’t stick around for the evening sessions. It’s the first day of the conference though, so people may have more energy than I give them credit for.

My session is also the only Night School session in English. I’m not sure if that’s good or bad for attendance. A few months ago I was rereading some of the Richard Feynman books (Surely You’re Joking… and Adventures of a Curious Fellow). In one of those, Feynman tells a story of teaching in South America and taking the time to learn Portuguese well enough to give the presentation in Portuguese. I should have taken inspiration from this and at least learned an introduction in German, but for some reason, German has always been a difficult language for me. I once gave an introduction to a class, including a short personal bio – all spoken in Japanese, and I could do at least as well in French (both languages are a bit rusty, but I’m sure I could pull it off). In German, however, I’m lucky if I can remember thank you and please. I’m not sure why I have the block, but I don’t worry about it very much (pretty much never except for when I’m in Germany).

The advantage to giving the talk in English is that anyone attending from outside of Germany who wants to attend one of the sessions will be forced to attend my session (cue evil laugh…). I doubt, however, that there are many (or more likely, any) folks attending who do not speak German except for me and a few other speakers, so I’ll have to win them over with content.

Oh yeah – after 350 words about my talk you’d think I’d have mentioned what it’s about. The title of the talk is “Testing at Microsoft: Past, Present, and Future”. I was never 100% happy with this title, but someone must have liked it, because there’s a keynote on Tuesday called “Design Patterns – Past, Present, and Future”. My talk is me back up on the “let me demystify testing and software engineering at Microsoft for you” soapbox that I’ve been on for at least five years now. The book (HWTSAM) was part of that soapbox, but so are many other talks, blogs, and articles over the past several years – as well as the slowly progressing MSDN Tester Center.

I think there’s a lot of mystery about how software is made at Microsoft, and how we manage and run our huge(!) teams. Even with HWTSAM, knowledge of testing at Microsoft is still largely hearsay and myths, and for some reason (nobody has ever asked me to explain testing at MS), I keep dispelling myths and wondering where people get their ideas (no, testers are not paid one pay grade lower than developers, and our testers do actually test software (which is slightly different than “only writing tools and automation all day”)).

This talk has some background on how the test role evolved at Microsoft, and what kinds of people we look for when we hire testers. I’ll talk about org structures a bit (b-o-r-i-n-g, but gives some context of how we manage groups like Windows that have well over 1000 testers). I’ll talk about strategies and tools too. I’ll also talk about some of the newest stuff we’re doing – including my take on customer focused test design (which I didn’t get into at all in my webinar last week), and some discussion of productivity games (aks “serious games“), and how we’re using games at work to increase productivity and quality.

But here’s the really important part. The session (if you did the time math above) is 90 minutes, but I didn’t prepare anywhere near 90 minutes of material. I’m not a slacker, I did this on purpose. When I give talks like this, my goal is to present “just enough” information to get people to ask questions. If you’re attending this talk (or any version of it in the future), it’s your opportunity to ask any question you want about testing or software processes or anything else remotely relevant to Microsoft or me. If I fill the time with too many charts and dense slides, you’ll be way too bored to ask questions at the end, and neither of us will enjoy the talk as much.

I am going to work so hard to make this an interactive session, that I will be disappointed (and perhaps slightly ashamed) if I get to my last slide any time before 8:05 (8:10 is my target). A larger crowd will help, but great questions will help even more.

We’ll see how it goes.

Follow up: I was happy with the way Monday’s version of this talk went. People asked a lot of good questions and seemed engaged. I went two (maybe three) minutes over time (sorry again), but covered a huge amount of “stuff” about making software at Microsoft.

I’m sitting in a hotel room in Munich. I’m slightly jet lagged (I’ll recover by tomorrow), but feeling better than I’ve felt in a week. I’m feeling so good, in fact that I have an abundance of "things" I feel like writing about.

If you don’t want to read a non-testing related post (or partially related at best), go ahead and skip this post – and maybe even the next one. I’m not sure where it’s going, but if I don’t start dumping these random thoughts out of my head, I may never get them out.

I’ll back up to last Sunday. My slides for the STP webinar (recording here, slides here) were due the previous Friday, but my rockin’ day-job was taking up a ton of my time, and all I had was an outline. You should note, that while this post isn’t about how to make a presentation, how to put together slides, or even procrastination for that matter, a good outline is the core of a good presentation. If you have an outline and notes that cover your main points, /and that tell a story/, putting the slides together isn’t much more than busy work. I could relate this to a story about arranging My Favorite Things for studio orchestra, but I’ll save that for another day.

I played soccer Sunday afternoon and had a pretty good game. After the game, I headed into work (my wonderful wife knows that if I need to crank out work, my work office on the weekend is the place to be. It’s quiet, it has a comfortable chair and large monitors). I arrived at the office around 6:30 (most buildings at MS have showers, so I was able to freshen up), and finished up the slides just before 9:00. I was tweaking, poking, adding, deleting and a bunch of other stuff I shouldn’t have been doing after a two hour slide session, but common sense prevailed and I decided it was time to head home. I saved the slides, then I reached into my gym bag to have another drink of gatorade before leaving. On a side note, I can’t prove this, but I’m 99% positive that the gatorade we drink today is absolutely nothing like what I drank when I was a kid. Now, I know that it came from powder, and we mixed it with water, but the stuff today is way to sweet for me.

So anyway…as I took the last swallow, I noticed a tiny little tickle in my throat. I’ve had the same tickle many times before, and it almost always means that some sort of cold his heading my way.

The next morning I woke up with a painful sore throat, but other than that I felt fine. I went to work, polished up the slides and sent them off to STP, then went through the rest of my typical work day. Tuesday, I still had the sore throat, but was feeling pretty run down too. I had a big chunk of stuff to do, so I made it through half a day then headed home for "a nap".

I went to bed Tuesday afternoon around 3:00, and didn’t /really/ get out of bed until Thursday morning. I was in bed so long that I had to get up and walk just to relieve the back pain from sleeping so much. I’d wake up from time to time thinking "I must be better by now". I’d sit and read while waiting to see if I felt strength returning. One time I was so "strong" that I could stay awake for almost 45 minutes before sliding back under the covers and drifting off again.

I’ve had many sore throats in my life (I got strep several times when I was a kid), but this one was special. Take a moment and swallow right now. Pay attention to the way the muscles in your throat contract, touch, and open as you swallow. Now, imagine that while you were sleeping, someone grafted a piece of sharp granite to your right tonsil (if you don’t have tonsils, imagine they wedged it into the hole where that tonsil used to be). Now, every time when you swallow, this piece of granite cuts into your throat. I’ve never wanted to not-swallow more in my life.

By Thursday morning, I felt well enough to take some cold medicine and go give a webinar (luckily I was muted when I started singing a medley of Colin Blunstone hits). I had a few other meetings to get through on Thursday (these things happen when you’re about to leave the country for nearly a week), but made it home by 3:30, took a nap (only 2 hours this time), had dinner, packed, then went back to bed.

I was still feeling out of it Friday morning, but could tell I was on the way back to normal. As of today (Sunday morning in Munich), I feel pretty good and should be 100% by tomorrow.

I should mention for those of you who don’t follow me on twitter, that my article Career Paths for Testers Using Personas is on StickyMinds (on the front page this week).

Some more of my thoughts on Personas are here and here.

A month or two back, Amazon began sharing BookScan information with authors. I can now see  how many copies of How We Test Software at Microsoft are selling (US only, and Nielsen BookScan only tracks the large-ish book sellers) – but it’s still sort of cool.

I can’t help but wonder who in Houston bought a copy, or why 4 copies sold in Washington D.C. last month.

I can also track ranking (only as far back as October 2009, I’m not as interested in sales ranking (the book seems to bounce quite a bit, but it’s still interesting to look at history. Sales rank from October 2009 to now looks like this:

Sales have been steadier than the sales ranks, and as such I expect that Amazon sales rankings are not based entirely on actual sales (links, page views, etc. all contribute (I think) to amazon rating).

Anyway – not sure what to make of it, but I’m pleasantly surprised to see that the book is still selling steadily. I think if we sell about a bajillion more copies, I can justify a second edition.

Twenty-eleven is kicking off with a flurry of activity for me – I think by the time I breathe again, a month (or two!) will have flown by.

First off, I have a short article showing up on TechWell next Monday. I hope I can talk Joey into letting me write somewhat regularly in the coming year, but for now, I’ll settle for the one article.

I’m giving an MS internal talk on “Tester DNA” on Tuesday the 11th (2:00pm, 40/Steptoe for those who can come – I assume it will be broadcast on resnet too). I expect to piss a lot of people off with this one, so it should be a good time.

I’ll be giving a webinar on Test Design for STP on Thursday, January 20 at 10:00am PST (more info here). I’m still figuring out exactly what I’m going to say for this one, but I promise it will be interesting.

On Friday the 21st, I’m flying to Munich for OOP. I’m honored to have been invited to give two presentations at the conference, and I’m looking forward to meeting some new people (and drinking German beer).

And in between all of that stuff, I have my day job. We shipped a few months ago and are in the middle of planning and making some fairly large technical moves (e.g. changing our test automation and distribution systems). I’m probably spread a bit too thin right now, but I’m confident I’ll end up prioritizing and giving each project I’m involved with just the right amount of push (personal kanban FTW!).

And finally – I am planning another webcast (using Lync) in February on career tips for testers. I’ll post more details in a few weeks. Since I can’t please the whole world, I’ll go at whatever time / day is suggested more than once in the comments section.

Happy new year.

I’m taking some time off this month from work, and think I’ll take a break from blogging too until 2011 (I reserve the right to change my mind if something really interesting comes up). It’s been a pretty good year for me, but as I reflect, it was a busy year too. Here are some highlights.

I began the year by blogging about my first 15 years at Microsoft (part 5 is here). I knew at the time that I was going to leave my position in Microsoft’s Engineering Excellence group, but didn’t know where. In the end, I had several offers, including a few that would have had me leave Microsoft.

However, by the end of February, I had a new office, and a new job in the Office Communicator (now Lync) team. I’ve had a blast on this team and have enjoyed every minute of it.

In April, I gave a STAR keynote for the first time. The talk seemed to go well, and the feedback form comments and scores (9.1/10) were all positive.

Over the summer, I took a short vacation with my family to Toulouse, France. At least I planned for it to be short, but when we got to our house, it was so nice that we decided to stay – for almost a month. I worked for the last two weeks from Toulouse and had a good experience learning how to work remotely. I was hit with some bad news the day before we returned home. My dad had a stroke (his second) and was hit pretty hard.  I spent most of the next two months commuting the 100 miles between Microsoft and my parent’s house helping out where I could and delving deeper into the world of remote work (fortunately, I work on a product that makes it easy to do this).

In October, I did a brief North American tour and hit Portland for PNSQC, then Toronto for TesTrek. I also gave two informal webinars: one on test strategy, and another rehashing my PNSQC talk (video here)

Microsoft Lync also shipped in October – my first shipping product in six years. Overall, not a bad year. There’s plenty more that I did – and a ginormous amount of things I wanted to do but didn’t get a chance. I hope that 2011 brings even more challenges, fun, and opportunities for success.

More in January.

Amazon just started sharing Nielsen BookScan reports to authors. Since I don’t get commissions for How We Test Software at Microsoft (hwtsam), I don’t generally see sales numbers, but I’m happy to see that people still buy the book (note for internal readers – orders through msmarket do not reflect in the BookScan ratings). Anyway – it looks like the book sold a whopping 32 copies last month. That’s certainly not a big number, but it’s sort of nice to know that people are still taking a chance buying a book about testing at Microsoft (also note that BookScan numbers are US only).

Today is the two year anniversary of the release of hwtsam. I wrote a post a year ago to commemorate the one-year anniversary, but I don’t think I’ll look it up to link to it because that would just make me too afraid to repeat myself.

I should take this moment to point out that Michael Larsen is in the middle of a chapter by chapter review of the book. He’s doing a fantastic job summarizing the book and sharing examples. Thanks Michael.

I remain proud of how the book ended up. Yes, it could have been better, and yes, I do find it slightly ironic that a book on testing was released with a small handful of typos, but I do believe the points still get across.

The book was a snapshot in time – ok, maybe not a snapshot – more of a window. If we wrote the book today, certainly, some of the subjects would change (and certainly my bio), we’d have more stories (or at least more current stories), but I’d guess that most of the subject matter would be the same – and I think that’s a good reflection on our choices for which areas of Microsoft testing to highlight.

The stories were my favorite part of the book. I’m sure this is partially because I didn’t have to write most of them, but I love books like "Writing Solid Code", and "The Pragmatic Programmer" that have stories and anecdotes buried within the prose and wanted to try something like that. I know that different people find value in different things, but I hope that some of the readers found value in the stories as well.

I recently put together a case study for an upcoming book containing a collection of test automation case studies (more details on this book when they’re available). In my search for an "interesting" case study to document, I came across dozens of really cool things going on within the walls of Microsoft that people wanted to share. My first thought was "come on folks – don’t you have a blog or something where you could share this stuff?" I’m not ready to shepherd another book yet, but someday when I find myself with a few spare months of time to blow, I may consider putting together a book of stories and anecdotes only about testing at Microsoft. We’ll see.

I speak at our new employee orientation periodically. I’m supposed to give an inspirational talk about my product or something I’m passionate about. I always talk about software quality and why we need the new people to speak up and use their fresh eyes to improve our company – this holds true for the software folks as well as the sales, legal, marketing and HR folks in the room. After a talk last spring, a young woman – a new college hire, tracked me down in the back of the room following my presentation. She asked me to sign her book and told me that she discovered the book in college and knew as soon as she read it that she wanted to be a tester at Microsoft. That situation has happened a few times since then, and the feeling of satisfaction on accomplishing one of my big goals in writing the book is huge.

In closing my annual Happy Birthday to hwtsam post, I want to say thank you again to those of you have read the book, supported the ideas, or even taken some of what we talked about and applied it to your own work or own organizations. You all made the effort that went into the book worthwhile.