What do you do when your manager | project manager | dev manager | other leader asks you for some bit of data that you know is useless? Say that they want something like code coverage progress and test pass rates, but you know that those metrics aren’t truly useful.

Side Note: waitaminute – some of you may be thinking those aren’t useless at all, The way I see it, coverage is only interesting in how you use it to discover potential holes in testing. Test pass rates aren’t interesting to me at all. I can have a 99.99% pass rate, but if the one failing test “accidentally” erases the customers hard drive, I’m far more worried about that single bug than the pass rate. Even if the pass rate is 95%, what I’m most worried about is what types of bugs are in that failing 5%

But people will still ask for those things, because they think they are useful. So, what do you do?

One approach would be to tell them they’re stupid and that you’re not going to give them useless data. Give that a shot and let me know how it goes.

On the other hand, you could just give them what they want – give them test pass rates and coverage information, and go back to work. that’s easy. (note – looking at these two examples is an application (or variation) of my “find the middle” technique).

What you want to do, is give them what they want, but most importantly, give them what they need. The coverage report gets a lot more interesting – and applicable if you include, for example, a list of tests you’ve added as a result of analyzing code coverage. It’s ok to share the test pass rate too, but include breakdown of the bugs causing the failures. You’ve given them what they asked for (pass rates), but you’ve steered the conversation towards risk (what they need). For bonus points, you can look for ways to intermingle the two requests (e.g. are you finding new bugs when adding tests discovered in coverage analysis).

One of the keys for success here is understand (or at least guess) why they want the data in the first place. They probably want some answer – project status, test progress…something. Figure it out, and give them whatever data you think they need to get their question answered. If you’re off track, reload and try again. You will find the right formula. The easy path is to give people what they want, but if you’re serious about improvement, start giving people what they need.

Should you automate everything, or nothing? Should you test everything, or nothing? How about leadership – should you dictate every detail of what your team should do, or give them no guidance at all. The answer for all of these questions – as you’d expect, is “somewhere in the middle”.

In my experience, most people handle the “how much” question when dealing with a range of potential solutions by starting with a reasonable mid-point and working from there – e.g. “let’s automate half of our tests”, or, “We’ll test what’s most important”, or, “I’ll give my team some guidance, and then give them some freedom in how they deal with the details.”

Those options are reasonable, so the technique seems to work. It, in fact, does work – but I think it can be better.

A brainstorming technique I use (someone please tell me if I’ve inadvertently stolen the concept) is to first spend a reasonable amount of time focusing on the extremes – because often, some great ideas for “the  middle” comes out of that brainstorming. Think, for example, what you’d do if you tested everything (yes, I know, impossible, but think about it. You’d likely use an army of vendors, need some sort of coverage metrics, etc. Then think about what you’d do if you tested nothing (you may have developers own unit and functional tests, and would rely on customer feedback for scenarios, etc.). In the end, there may be something you take from both brainstorming sessions when you figure out what “the middle” looks like.

How about we try another example. Let’s say you are testing application compatibility with version 2.0 of the “Weasel” operating system. There were 100 applications written for Weasel 1.0, and you have copies of all of them. How many of those applications do you test? If you go straight to the middle (which, again, isn’t a bad choice), you’d probably prioritize the apps by sales numbers and test the top n number of apps based on how much time you have. Not a bad solution, and one I’d feel comfortable bringing to the team leaders.

But let’s think about the extremes for a bit. What would testing all 100 applications look like? We’d definitely need to outsource the testing – but in order to do that, we’d need some clear directions on what “testing” an application entailed. We could write separate notes for each application, but there are probably come up with something generic (install, uninstall, copy/paste, print, major features, etc.) that could work. This solution is certainly going to be too expensive for Weasel management to approve, but we’re just brainstorming.

Now think for a while what it would be like to test none of the apps (and not piss off customers). Well, if none of the programming interfaces used by the apps changed, they’d all probably still work. But this is Weasel 2.0, so of course we’re going to tweak the APIs. So, maybe it’s possible to profile the APIs used by the Weasel 1.0 apps and diff that against the APIs we’re changing in Weasel 2.0 and then develop an API test suite that ensured API compatibility. There may be something here…

Of course, neither of these solutions is the right answer (nor are my brainstorming sessions complete), But I’ll bet that if you try this approach the next time you’re dealing with a range of possible solutions, you’ll come up with some new ideas on what you may choose to do “in the middle”.

Phil Kirkham confirmed his birthright as a tester by asking the epitomical tester question regarding my last post.

Why?

To be fair, he actually asked:

quite a schedule – so what do you get out of it ? Or conversely, if you didn’t go to these events what do you think you would be missing ?

This is something I think about myself (and occasionally need to explain to my management chain, as time at these events is time away from my day job), and I thought there may be enough here to discuss to merit another post (you may, of course, disagree).

The first event I mentioned, the UCIF event, is pretty much part of my job, So I suppose what I get out of it is that I get paid. I think it will also make my overall ucif work easier, so it’s definitely a good thing for everyone involved (except, perhaps, my family).

The other gigs (PNSQC, German Testing Day, Intel, and the Webinar) are where the answer is (possibly) more interesting. If I didn’t go (and present) at these events, I could still get my day job done (and in fact, you could argue that I’d be more effective at my day job because I wouldn’t be missing work). If I were a consultant, I would be using the events to drum up business, but that’s not the case here.

I think the best explanation has two parts. The first is that I sort of like engaging in the test community. PNSQC is a no-brainer since it’s close by and I love Portland (thus it’s the only conference where I’ve asked to present (via abstract submission) in the last several years). But PNSQC – and all of the other gigs on my list are great opportunities to meet testers and discuss hard problems in testing (and some solutions for these problems). I like talking about testing, and talking with non-MS testers gives me (I think) a better perspective on what’s happening in the world. I certainly don’t agree to every speaking offer, but I do try to accept a few opportunities every year. It just so happens that my opportunities this year are all consolidated into a five week period. I may do something in the spring, but as of now, I think the fall flurry will be it for me for a while.

I also like to think there’s value to Microsoft as well (at least my manager and I do). As I thought when writing HWTSAM, I think there’s value in sharing what happens at MS with the worldwide testing community. I often say, “in the absence of information, people will make stuff up”. I try to reduce the amount of stuff people make up  by sharing what I can about how teams at MS approach the practice of making quality software at a large scale. Sometimes it works, sometimes it does not – but I feel the effort is worth it.

So, what do I get out of it? I’d like to think I get better software testing – either through ideas I share with others, or by refining my ideas based on feedback, or brand new ideas I can bring back to MS. I think it’s worth it.

I just got back from (yet another) vacation with the family – it’s been fun spending some extra time with the family this summer, but I think we’re (almost) done now, and it’s time to get back to work and to start thinking transitioning the kids into sleep schedules that accommodate school hours.

With the approaching end of summer comes my inevitable fall travel schedule.

• I’ll be in Austin, Texas on September 13 & 14 to attend the UCIF members conference.
• I will be at PNSQC in Portland on October 10 & 11. I will be giving a presentation on Customer Focused Test Design.
• I’m giving a webinar on October 12 – details coming soon.
• I’ll be in Israel October 26-28 speaking at Microsoft and Intel. This is my first trip to Israel, and I’m quite excited about this trip.
• I’ll be giving a keynote at German Testing Day on November 9. I met some of the folks who put on this event last year, and I’m utterly honored to have been invited to speak at this event.

Considering I actually have a day job, this schedule is probably a bit rough, but I’m excited to have these opportunities to speak and to meet testers from around the world.

If you’re in the area of any of these events and want to meet up, please let me know (or come talk to me at one of the sessions if you’re attending).

I wrote about using virtual machines for testing in hwtsam, and gave a talk at STAR West a few years back on using virtual machines for testing. What I remember most about the talk was the group of VMWare employees sitting about 10 rows back (I think to ensure that I didn’t say anything bad about them – I didn’t). In fact, I’m a fan of virtual machines no matter where they come from – they’re incredibly useful, and in many cases, and under used productivity aid.

One point I seem to forget to mention when I’m talking about virtual machines is how convenient they are beyond testing purposes. I wanted to share a recent experience of mine, but there’s a bit of a story leading to the punch line, so feel free to skip ahead a few paragraphs if you’re busy.

I have three computers in my office at work. My main “dev / test box” is a win2k8 server (for reasons beyond the scope of this post, we build our product only on server machines). We have fancy tools that let us build and deploy (for testing) quickly on this single machine, and in order to not break the consistency of this niceness, we never, ever install any shipped office bits on these machines – only bits that we generate through the build process. The consistent use of machine and build configurations eliminates nearly all of the “it doesn’t work that way on my machine”, or “my build is broken” issues.

Because real users don’t use our dev machines, I also have a “test machine” – which I use for (surprise, surprise), testing. Because I’m working on the next version of Office, that’s all it runs most of the time.

Then I have my laptop. Since I use it for everything else, it runs (usually – but see below) office 2010, and I use it for email, docs, etc. About a month ago, I needed to install our pre-beta bits on my laptop. In general, we can run our new stuff side by side with the shipped stuff, and it works fine. However, in a long bout of yak-shaving one day, I had to uninstall all versions of Office from my laptop. But I still needed the latest bits to reproduce a bug I found, so to save time, I just installed the new pre-beta bits.

You see where I’m at now. I don’t have any machines running shipped office bits anymore. While the pre-release apps all work quite well, they’re a bit…unrefined. I’m a big fan of ‘eating my own dogfood’, but sometimes I need the confidence that comes from using something a bit more baked.

I almost started to install the shipped office bits back on my laptop when I realized two things. The first was that I was extremely worried that I’d spend a few hours every week tweaking things on my laptop – installing, and uninstalling in order to clear up weirdness, or keeping things in sync switching between different versions of the same app. I want to continue running daily builds as much as I can, and knew I’d end up switching between different versions of the same application all the time, and I was worried about the time hit. The second thing I realized was that a virtual machine could be a nice solution for me.

It only took me an hour or two to set up a hyper-v vm (hosted on my “dev box”), install windows, office 14, and sync documents with my live mesh account and I had a vm that acted so much like a ‘real’ pc that after a month, I still forget sometimes that it’s not a real machine. I use our prerelease bits for nearly everything, but can rely on the vm (which I connect with over terminal services) when I feel the need to work with shipped software – for example, my vm currently has a copy of word open with my pnsqc paper. It’s cool, because I was working on it last night (connected from home), and just left word open. When I logged on to the vm this morning from work, it was right where I left it. It’s so damn convenient, in fact, that I can’t imagine not having a “work” vm anymore, and I can see a point in the not-too-distant future where we all use virtual machines in the cloud more than physical computers.

– written and published from a virtual machine

My interlude is over, and I’m back to blogging – at least that’s the plan, and I don’t see any reason why I won’t be back on the blog-waves on a semi-regular basis.

My summer was crazy with work. I probably let myself get spread too thin, and I paid the price of context switches and deadlines and delivered what I planned to deliver. One nice thing that happened amidst all of this work is that I had a chance to practice and polish my personal kanban process. I’m a big fan of pkb and should probably share my approach to the tool sometime.

But I took a break for more reasons than being busy. Every few months, it seems, I seem to have a personal crisis about the future of software testing. I sort of find it ridiculous that there are so many self-proclaimed experts in a field that is at best barely defined. Everyone seems to have a definition of what testing is, and what testers do. I think that’s perfectly fine…until testers begin to discount (or mock) the testing work of others just because what their definitions of roles and approach don’t match. I bet what I do is different than most of you – but we’re probably all still testers (of some form)and I expect we can find some value in the diversity of our approaches.

Then, after reading one too many articles from testing “experts” that failed to grasp many of the basic concepts of what they were writing about, I began to fear that we’re taking a step backwards for every step we advance. Rather than wrestle with so many things outside of my control, I decided it was time for me to focus on what I could control and give myself a break from the testing community. I love you all, but sometimes you drive me crazy.

I didn’t plan to solve any of my internal strive during my time away, and I didn’t. But – I think I’m better equipped to focus on what I can control and continue to share what I find valuable in my world. I still believe in a bright future for software testing, but we need more open, seeking minds to get us there.

As you may have noticed, I’ve had to take a bit of a break from blogging. I’ve kicked out at least a post or two a week for a few years now, but I’m not dry on ideas, and I definitely want to follow up on my recent Test This post (before a follow up becomes irrelevant).

The challenge is that my day job is kicking my ass. I expected the UCIF test and certification work I’m involved with would take about 20% of my time (actually – they told me to expect 20% – I was hoping to get away with a bit less), but lately it’s been making a much bigger dent in my work day. Currently, I’m preparing a progress report for the board, and a members only webinar for later this month, along with trying to gain some ground on the roadmap for the working group.

The Lync team is bearing down on an internal milestone and there are a lot of loose ends (both technical and pseudo-political) to tie up in the next several weeks. Yearly performance calibrations are going on at the same time, so along with my normal levels of coaching and mentoring, I’ve had a few extra sessions of tester therapy in the past weeks.

And then there’s the not-really-work stuff that I still need to do. Top of the list is finishing my PNSQC paper, and I’m also juggling and negotiating some potential speaking engagements for the fall. For the first time ever, I’ve also had to start saying no to some internal speaking requests (sorry – if you read my blog, I hope you ask me again later this summer).

So – once I find a light at the end of the tunnel (or when the day job finally breaks me), I’ll be back.

The “Test This” game pops up frequently in testing circles. Sometimes it pops up in the “How would you test a …” variety – e.g. How would you test a stapler. Participants in the game are expected to come up with functional tests (can it staple), as well as “out of the box” tests like, “what happens if I drop the stapler off the top of my house” (I guess it’s a form of a reliability test). These are fun games, and while I suppose they do force some creativity, I don’t think they really have anything to do with testing software.

There are better flavors of the Test This game. Sometimes, testers play the Test This game with real software. This form has huge potential, but never really delivers, as the software under test is typically a bug-infested piece of crap that anyone with a pulse could find bugs. Bugs are part of my problem with this form of Test This – the goal of this game is to find bugs – not to test or learn testing. That’s an important distinction for me – the goal of testing isn’t to find bugs. Finding bugs is merely a side-effect of the testing process. I fear that these exercises encourage testers to dive into bug finding before thinking about how to test. To be clear, I’m not talking about big-upfront-test-design – but a holistic approach to testing.

As a side note, I really like what Weekend Testing is doing with the Test This game – sure, they pick some buggy software sometimes, but they also pick real world apps and discuss test design (at the very least, in the debrief).

Here’s an example (I’ve use this one often). This is the Microsoft Word font dialog from Word 2010. If my memory serves me correctly, this version may be less complex than the previous font dialog.

Note: I’m choosing this dialog because it’s interesting. I don’t know who tests this, and I bet they will have much better ideas on how to test this than me, but that doesn’t matter.

Ignoring the advanced tab and the functionality of any of the buttons other than OK or Cancel, how would you test this?

You can spend as much time as you want (or use as little time as you need). I don’t think I care about the actual tests as I care about how you decide what to test.

As always, you don’t need to actually reply to this post – just do the exercise if you want. I’ll write up some thoughts in a week or so and see if they make sense (I don’t think I have any idea where I’m going with this.

A former colleague of mine (a former manager, to be exact), recently blogged about the plight of the SDET as second class citizens earlier this week. I was going to reply to the post right away – then I was going to reply in a blog post, but in the end I decided to wait a few weeks to collect my thoughts and see if I could come up with something better to say.

So here goes.

The first time I read the article, I thought, "Eric is living in the past – this just isn’t the truth anymore." Career paths and respect for testers is something I’ve been working on at Microsoft for a long time, and I know where we came from. We have over ten times as many testers at the "Senior" and higher levels (i.e. they get paid huge bucks) than we did four years ago. I work on a team where the testers are absolutely fantastic, and where levels are roughly matched between test and development (we actually have more senior testers than senior developers on our team). Sure, there are pockets of "trouble", but 99% of the time, programmers and testers live on equal ground. I have colleagues in other groups in similar situations, so although my group is exceptional, it’s definitely not unique.

But because I live in such a great world, I forget how often that isn’t the case. I’m reminded that on some teams, testers aren’t matched level-wise with their programmer peers and don’t have "first class’ respect. For every team like mine, there’s another team where testers are treated like crap and are looking for an escape or solution.

One thing to keep in mind that testers are promoted just as often as programmers (and other disciplines) at Microsoft (I’ve done the research, and I’m sorry to say that there’s no conspiracy theory in play here). The biggest reason we have (in general) higher leveled programmers than testers is that many testers move to programming (development, as we call it) roles before they get to senior levels in test. To me, that’s fine. If you want to be an "SDE" at Microsoft, and you spend a few years as an "SDET" to get your feet wet, that’s fine with me. I think testers make great developers – especially the one’s that didn’t want to be testers in the first place. The last person I want taking on big responsibilities or hard challenges on a test team is someone who doesn’t want to be testing software (or doesn’t care about testing software).

Eric goes on to give a developers overview of what test can do and (I think) makes the real point of his article.

Then the comments came and I had another thought.

Are testers just a bunch of whiners?

 

I can’t imagine another profession where people complain so much about the crap they have to deal with, yet won’t quit because they “love their job”. Yes, there is testing, and there is dealing with people, but both are attributes of a professional, so I don’t get it.

Here’s a tip – you don’t "get" respect, you earn it. In my experience, testers "earn" respect by being credible, showing good results, and making good decisions. You don’t "get" respect from your "level", or by demanding it. Seriously, stop whining and figure out how to earn respect.

Let me also put some blame on the test managers. It’s your job to make sure your team has the tools (including mental models) to do their jobs (including relating with their dev peers). If they can’t do their job, it’s probably your fault.. If your team is bored, fix what your team is doing. If the job is manual (IMO, manual scripted tests are a black mark of idiocy on the testing world), either automate it (if it makes sense), or stop doing it completely. I know, I know- your situation is “unique”, so your team has to piss off highly paid testers with programming degrees. I”ve heard the “unique” line a zillion times now, and it’s simply not true.

In all seriousness – I don’t care whether you work at Microsoft, Wal-mart, or Chik-Fil-A. If you don’t think you get enough respect at work, try to earn respect. If that doesn’t work, quit and get a new job. Life’s too short to have a job where you aren’t challenged, motivated, or respected. But please stop the whining – it’s ugly and does nothing to help.

you with me?

I think it’s been a long time since I took more than a week between posts. I was busy at WAT2, and then busy catching up from being busy at WAT2. I’ve had a busy travel schedule this year (a plane trip every month so far this year), and although Durango is closer to home than most of the places I’ve been, it’s still travel, and it still takes time. I’m staying off of planes for the foreseeable future. I had an opportunity to travel to China later this month, but I had to turn it down. I’m on the hook for a web cast during the trip – and I just don’t feel up to more travel right now. I really, really want to go to China sometime, but it just isn’t the right time for me to make that trip.

I had a great time at the Writing About Testing conference. Chris McMahon is a wonderful host, and the Durango weather couldn’t have been more cooperative. I enjoyed the variety of conference activities – from full on presentations to lightning talks to writing activities, I was fully engaged for the entire conference. I met some new people, met some folks I’d known through twitter, and reconnected with others.

One of the first writing exercises of the conference was to pair up and take a shot at writing the traditional five paragraph essay. The 5PE is a staple of writing, and Chris spits these out faster than he can come up with blues bass lines. I think most of us learned the 5PE at some point, but I, for one, have drifted away from the form over the years (for better AND worse), so this was a welcome exercise for me. In addition, the fact that we were pairing on the writing added another dimension. I paired with @m8ttyb (aka Matt from Mozilla), and we came up with a draft for what I think will be a publishable article. I think the pairing format is an interesting opportunity to share writing styles and writing methods. My only regret with the exercise is that I worry I pushed my own methods on Matt too hard and didn’t take the time to learn what works for him. If this article works out, I’ll see if he wants to write another one some day, and use that opportunity to listen more.

After the conference ended on Saturday, a handful of us went to Carvers Brewpub for dinner and beers. Somewhere in the middle (or near the end) of those beers, in a story that I don’t think it will ever be possible to retell, "expectpants" blossomed as the phrase of the conference (some participants may disagree, but we’ll catch them up eventually). I’m not exactly sure what’s going to happen with expectpants, or if the full story will ever be told, but it was profound enough at the time that I felt it was worth sharing.

Last I heard, Chris is unsure if there will ever be a WAT3, but I figure if he’s not up for it, that if 15-20 of us want to "randomly" descend on Durango for a weekend in May next year that  he’d be welcome to show up at whatever informal gathering we put together.

Until next time…