For testers, being skeptical is generally a good thing. When someone says, “Our application doesn’t have any reliability errors”, I, for one, am skeptical. I’ll poke and prod and (usually) find something they haven’t thought about. There’s power in skepticism. Last year, I led a team of testers in performing code reviews of production code. My hypothesis was, that while developers perform code reviews thinking, “Does the code do what it’s supposed to do”, testers think, “In what conditions will the code not do what its supposed to do. You can insert the comment about testers being pessimistic (or overly pessimistic) here, but in general, the tester mindset is to question statements that seem…well, questionable.

But it’s easy to go overboard with skepticism. Time and time again, I hear good testers apply their skepticism broadly and liberally. Some (paraphrased) quotes I’ve heard recently include:

• “Model-based testing is interesting, but it doesn’t work in a lot of places”
• “I’m skeptical of static analysis tools – sometimes they have false positives”
• “Metrics are evil, because someone may use them to measure people”

I agree whole heartedly with each of these quotes. However, I worry that folks are throwing the baby out with the bathwater. Model-based testing (just an example), is a wonderful test design technique for stateful test problems. Although occasionally someone will screw up the value of MBT by claiming that it’s the only test approach you’re ever going to need, it’s just another technique (and the perfect technique given the proper context). Static analysis tools are also awesome, but aren’t perfect It’s good to measure some things too, but sure, one can screw it up.

I’m trying to think of anything in my life that works perfectly in every situation, but I’m coming up empty. I run into situations nearly every day where someone has a good idea that will obviously work most of the time – but not always. Given these situations, we could just send them back to the drawing board telling them, “I’m skeptical of your approach, because it won’t work in situation z”, but it’s probably a better idea to have a conversation about the limitations, understand where and when the approach may fail, and discuss mitigation or workarounds. Instead of throwing out the idea of running static analysis tools because of the potential false positives, discuss the false positive problem. Find out what causes them. Tweak the configuration. Do whatever you need to do to ensure the value of the approach.

Over the years, I’ve found value in some pretty stupid approaches. It seems that we should be able to find more value from some the ideas frequently discounted.

Even if we’re skeptical.

Chris McMahon’s latest post (Just Fix It) proposes that as far as bug tracking goes, the best course of action is to skip the “tracking” part of the workflow and “Just Fix It.” I’m a huge fan of this approach and think that for the most part, tracking a large number of bugs in a big fat bug system (and often overemphasizing the church of the bug curve) pretty much encourage a test-last / test-quality-in workflow.

I see this concept come up frequently, and I’ve noticed a bit of a trend. Teams that follow Just Fix It love it. Teams that prefer to fix bugs later are sure that the concept won’t work for their team, and that they need the curve and tracking data in order to ship their (undeniably unique) product. As a side note, one fun thing I’ve talked a few teams at Microsoft into doing is to pair on bug fixes – when Joe-Tester finds a bug, he Tells Kathy-Developer about the issue, and then the two of them pair-program the fix. I could write a whole post on why this is so cool, but I’ll leave it at that for now.

In short, I’m a huge fan of Just Fix It, but as usual, the totality of overlap of agreement between Chris and I is about 93%.

For example, let’s say on Monday, Alex-Tester notices that the froobnozzle isn’t working. He tells the developer, who fixes the problem immediately. Tuesday, Beth-Tester notices that the froobnozzle doesn’t work when interacting with another part of the system. She tells the developer, who fixes the problem right away. Over the following days and weeks, a lot of problems are discovered with the froobnozzle. The froobnozzle, is, in fact, a piece of crap held together by the 1s and 0s version of spit and duct tape. A bug tracking system let’s you see (I hate to say this out lout…) trends of where errors are. Bugs don’t appear randomly sprinkled throughout a product – they tend to congregate in clumps. Knowing where the clumps are can guide further testing, or risk decisions.

Source control almost mitigates this concern, but unless you have a diligent comment policy for check-ins, you probably won’t be able to differentiate between a “I was adding new code or functionality” check-ins vs. “I was fixing shit I broke” check-ins.

But I won’t go as far as to say you need a bug tracking system. As Chris describes it, and as most people use them, a bug system is really a work-item tracking system anyway. If you track work items on post-its or notecards, bugs should work the same way. As far as trends go, I think a simple tic-mark system would work just as well. When you discover a problem with the froobnozzle, write it on the board and put a tic-mark next to it. When a component gets n number of tics, schedule refactoring time, or do a design review (or both). Alternately, look at the components with the highest number of tics during the retrospective or sprint planning and review those for potential re-work.

But other than that, yeah. Just Fix It!

Note: I would have posted this as a comment on Chris’s blog, but I, for one, find the comment forms on blogspot completely unusable.

I posted a few months ago regarding my move to using a virtual machine as my “mail” machine. I’m still a huge fan of that approach, but in my new job (the really cool one working on the Xbox team!), I’m running Win7 rather than server as my main machine. I looked into using an internal hosted VM server solution, but since the solutions I found are used primarily for testing, performance wasn’t quite what I wanted (or what I was used to).

Let me outline the problem. I don’t look at email or twitter or my rss feed all of the time – in fact, most of my electronic communication time occurs when I’m compiling / building (I could use that time to have a sword-fight, but I find it better to at least pretend I’m productive). I love to group the “non-essential” stuff together, and when I’m waiting on a build, it’s the perfect time to write email, documents, tweet (or sometimes blog).

The problem is, that when I’m compiling, my 12 cores are pretty busy. This makes reading email, or doing pretty much anything else that would like more than a few time slices of cpu power sort of suck.

I have two computers (a desktop and a laptop), so I have an option of using my laptop for those tasks while the compiler elves are taking over my computer, but given the choice of using a 24” monitor and a full size ergonomic keyboard over a 13” monitor and compressed keyboard, which would you choose? To be fair, I adore my laptop (Lenovo X200s) – but I adore it because it’s small and I can use it anywhere (including on an airline tray table while the monster man in front of me leans his seat all the way back for 8 hours straight). But it’s not a computer I like to write large documents or presentations on.

The solution, as many have already guessed, is to connect to my laptop via terminal server and use it from my desktop machine. The overhead of TS is small enough, that I barely notice a perf problem even at peak memory and cpu usage. The TS session even stretches my desktop to 1920×1080, so it’s a pretty sweet setup. All that is good (and somewhat obvious), but few things in life ever go without a hitch.

Despite how much I wanted to love the setup, I noticed that my connection would drop frequently – and then it would take a minute until I could reconnect. I powered through it for a day or two (I tell myself that I was subconsciously gathering clues). Then finally, I found the clue that led me in right direction. I listen to music on Zune quite a bit while I work (I configured terminal server to play audio on the server machine (my laptop), so I select songs from the TS session on my desktop, and listen through headphones connected to my laptop). So…I noticed that whenever my computer disconnected, I lost audio a few seconds later. The first few times this happened, I (ignorantly) assumed that because I lost the TS connection, I also lost the audio. Then it hit me that since the audio is playing on my laptop, I should never lose audio…unless the network connection was lost.

Losing the TS connection and losing audio were both symptoms of the same root cause (lost network connection) – the only thing left to figure out was why two computers on the same subnet were losing their connection.

I took a look at power management settings to ensure that my laptop wasn’t suspending, but all was good. My next step was to look at my adapter settings to see if the network adapter was powering down for some reason. Sure enough, there was a setting named “System Idle Power Saver” that was enabled. I disabled the setting, and I’ve only had one dropped connection all week.

This example may sound more like yak-shaving than testing, but I’ve always liked the trouble shooting aspect of testing. Going beyond “here’s a problem”, to “here’s what (probably) caused the problem” is, in my opinion) an important skill for testers.

Oops – my build is done – time to get this posted.