Michael Bolton noticed my last blog post and wrote a nice follow up outlining several factors that lead to testers getting lost in the weeds vs. finding great bugs. I had my own ideas when I wrote the post, but the breadth of suggestions from Michael, as well as from commenters on the original post were fantastic.
To me, two of those suggestions sing most true. First, Michael suggests that one problem is the overemphasis on proving functional correctness (several of his follow up points re-emphasize this). I couldn’t agree more. I worry a lot that testers are so focused on running through their short little lists of functional test techniques to “make quality software”, that they miss a lot of stuff right in front of them – or mis-prioritize what they see. Not coincidentally, this is something I’ve been thinking about (and working on) a lot lately. The discussion topic I’m bringing to the Writing About Testing conference is along the lines of "The End of Functional Testing as We Know It" – but I’m still working on a better title….as well as the essay to back up the idea (hold on Chris – it’s on my short list). My team at Microsoft is also looking at how we can lessen the emphasis on functional testing, and as we make more progress, I’ll be sure to share more.
The other bit I think has a huge contribution to this is something Joe Strazzere said in a comment
I think much of it is motivation. Some shops, and some testers, are very positively motivated toward higher bug count.
This is unfortunately too true. The emphasis on bug finding and testers who have bug quotas or who are paid or rewarded by the bug encourages testers to report anything in the world that they consider slightly out of the ordinary.
Now – this is where the judgment comes into play. Of course, if something has the potential to annoy someone, it’s a bug and we should report it..
Or should it? Look at the following (fake) dialog message.
This action has been completed. Please check your work, save any open files, then close the appIication
OK – now find all the bugs in that error message.
I can find at least four, but which is most important? Which bug do you put up on a flagpole as the one that must-be-fixed vs. the bugs you can log and then fix “later”? The answer, of course, is, "it depends", but we can talk a bit about what it depends on.
The most critical bug is probably the misspelling of the word ‘application’. If you look closely, you’ll see that the fourth letter ‘L’ is actually a capital ‘I’ (pronounced eye). Oops, that’s huge and we’ll need to fix it.
Next off is probably the missing period at the end of the sentence – but I’d look for consistency here before reporting it. In general, sentences in error messages end in a period, but I’d want to see if just this error was missing the period or if it was a preference that was more wide spread. If so, I could enter one bug to add periods to the end of all error messages. Unless, of course, I’m paid by the bug, then I would add bugs for every error message in the program (woo-hoo – I’m going to Disneyland!).
The others are more subtle. It may read a bit better to say, "…save any open files, and then close the application". How about the first sentence – doesn’t passive voice just grate on you. That must be a bug. Go ahead and insert a comment about needing someone capable of writing error messages on the team. What I worry about a bit – and what I’ve seen is testers so focused on finding any bug, that they find the passive voice and grammar bug well before they find the real bugs in this error message. That’s too bad.
If you’ve paid attention, there’s an underlying message of this post. Reading between the lines, I’m sort of saying that functional testing and finding bugs aren’t that important in testing. I’m somewhat ok with this statement, but I’m sure it will bother someone. My point is that if testers focus on making sure the software provides value to the customers, and let finding bugs (including functional errors) exist as a side effect of that focus, that good things will happen.
“why do we need a dialog box at all?” would be my first question and “remove unnecessary dialog box” would potentially be the first bug i opened
if we did need the dialog, then the customer value of fixing any of the four errors in the string is unclear to me so i would probably just open one meta work item / bug like “perform editing pass on resource strings” and include the sentence as an example
but you need to stop thinking like someone who isn’t paid by the bug :}
“I’m sort of saying that functional testing and finding bugs aren’t that important in testing.”
Hmm. I’m really rather uncomfortable with that statement, and I can’t quite put my finger on why.
If not to the testers, should functional testing and finding bugs be important to anyone? And if it’s not important, then I have to expect that some bugs won’t be found – and thus can’t be triaged, and can’t be fixed.
If you are saying that there is a bigger picture than just “find many bugs”, then I agree.
If you are saying that testers should use their judgement and decide not to write bug reports for things they notice which they believe to be bugs, then I disagree.
I’d love to read more about “The End of Functional Testing as We Know It”. Perhaps it will provide some comfort.
From a user experience, that fake dialogue is awful. At best, it’s extraneous, provides no real information, and condescends to the user.
I think I disagree with your premise that functional testing and bug-finding aren’t that important, but possibly just from the way you’ve elected to phrase it- in order to be successful testers, we need to be first and foremost advocates for the users of the stuff we’re testing. If we’re successful in doing that, we’re going to suss out the functional bugs- not automatically because we’re doing ‘functional testing’ because we’re not explicitly testing features in isolation, but rather holistically.
This may not seem like a huge difference, but I think in practice it makes the difference between “does the object to be tested do X, Y, and Z” and “in the course of doing X, Y, and Z, unexpectedly, the product does A and B.” The former is testing in extremis, the latter is testing something by being integrally involved with the focus on making it live well in the environment it will inhabit.
I was kind of hoping that I’d cause a bit more of a violent disagreement with the phrase ” functional testing and finding bugs aren’t that important in testing”. Instead, I’m sure people just unsubscribed to my blog.
The way I *really* feel isn’t phrased that much differently, but I’ll throw it out there to those that read comments:
“Functional testing and bug finding probably aren’t as important as many testers think”.
Better?
Much better. I can agree with this completely- unless the people who are the consumers of the products we test are closely involved with the design of individual features, they’re not going to know if the product works as per spec in n of m situations.
What they will know is the more qualitative stuff- “does this fit my workflow” and “is it consistent with behaviors elsewhere in the rest of the product/other products I use.”
Hi Alan,
I agree that testers should find critical customer impacting bugs rather than functional bugs. But in practice we cannot get this done right away. First of all, why are there still so many functional bugs in the products we are testing?
Verifying/checking that the product behaves as specified in the document is not testing at all. It is just checking NOT testing. If testers are catching more functional bugs, we need to perform a causal analysis to find out why. Did the developer perform the unit testing properly? Were there so many functional issues that the testing team had no time to find critical issues?
Regards,
Aruna
http://www.technologyandleadership.com
“The intersection of Technology and Leadership”
in general context, how do you define “functional”.
At times it boils down to the standards and processes teams have to categorize the bugs. Some teams say, “when a bug is opened, it is by default of type Code and of functional category” are they always of functional category? that is contextual and only decided on item by item basis.