Michael Bolton noticed my last blog post and wrote a nice follow up outlining several factors that lead to testers getting lost in the weeds vs. finding great bugs. I had my own ideas when I wrote the post, but the breadth of suggestions from Michael, as well as from commenters on the original post were fantastic.
To me, two of those suggestions sing most true. First, Michael suggests that one problem is the overemphasis on proving functional correctness (several of his follow up points re-emphasize this). I couldn’t agree more. I worry a lot that testers are so focused on running through their short little lists of functional test techniques to “make quality software”, that they miss a lot of stuff right in front of them – or mis-prioritize what they see. Not coincidentally, this is something I’ve been thinking about (and working on) a lot lately. The discussion topic I’m bringing to the Writing About Testing conference is along the lines of "The End of Functional Testing as We Know It" – but I’m still working on a better title….as well as the essay to back up the idea (hold on Chris – it’s on my short list). My team at Microsoft is also looking at how we can lessen the emphasis on functional testing, and as we make more progress, I’ll be sure to share more.
The other bit I think has a huge contribution to this is something Joe Strazzere said in a comment
I think much of it is motivation. Some shops, and some testers, are very positively motivated toward higher bug count.
This is unfortunately too true. The emphasis on bug finding and testers who have bug quotas or who are paid or rewarded by the bug encourages testers to report anything in the world that they consider slightly out of the ordinary.
Now – this is where the judgment comes into play. Of course, if something has the potential to annoy someone, it’s a bug and we should report it..
Or should it? Look at the following (fake) dialog message.
This action has been completed. Please check your work, save any open files, then close the appIication
OK – now find all the bugs in that error message.
I can find at least four, but which is most important? Which bug do you put up on a flagpole as the one that must-be-fixed vs. the bugs you can log and then fix “later”? The answer, of course, is, "it depends", but we can talk a bit about what it depends on.
The most critical bug is probably the misspelling of the word ‘application’. If you look closely, you’ll see that the fourth letter ‘L’ is actually a capital ‘I’ (pronounced eye). Oops, that’s huge and we’ll need to fix it.
Next off is probably the missing period at the end of the sentence – but I’d look for consistency here before reporting it. In general, sentences in error messages end in a period, but I’d want to see if just this error was missing the period or if it was a preference that was more wide spread. If so, I could enter one bug to add periods to the end of all error messages. Unless, of course, I’m paid by the bug, then I would add bugs for every error message in the program (woo-hoo – I’m going to Disneyland!).
The others are more subtle. It may read a bit better to say, "…save any open files, and then close the application". How about the first sentence – doesn’t passive voice just grate on you. That must be a bug. Go ahead and insert a comment about needing someone capable of writing error messages on the team. What I worry about a bit – and what I’ve seen is testers so focused on finding any bug, that they find the passive voice and grammar bug well before they find the real bugs in this error message. That’s too bad.
If you’ve paid attention, there’s an underlying message of this post. Reading between the lines, I’m sort of saying that functional testing and finding bugs aren’t that important in testing. I’m somewhat ok with this statement, but I’m sure it will bother someone. My point is that if testers focus on making sure the software provides value to the customers, and let finding bugs (including functional errors) exist as a side effect of that focus, that good things will happen.