The “Test This” game pops up frequently in testing circles. Sometimes it pops up in the “How would you test a …” variety – e.g. How would you test a stapler. Participants in the game are expected to come up with functional tests (can it staple), as well as “out of the box” tests like, “what happens if I drop the stapler off the top of my house” (I guess it’s a form of a reliability test). These are fun games, and while I suppose they do force some creativity, I don’t think they really have anything to do with testing software.
There are better flavors of the Test This game. Sometimes, testers play the Test This game with real software. This form has huge potential, but never really delivers, as the software under test is typically a bug-infested piece of crap that anyone with a pulse could find bugs. Bugs are part of my problem with this form of Test This – the goal of this game is to find bugs – not to test or learn testing. That’s an important distinction for me – the goal of testing isn’t to find bugs. Finding bugs is merely a side-effect of the testing process. I fear that these exercises encourage testers to dive into bug finding before thinking about how to test. To be clear, I’m not talking about big-upfront-test-design – but a holistic approach to testing.
As a side note, I really like what Weekend Testing is doing with the Test This game – sure, they pick some buggy software sometimes, but they also pick real world apps and discuss test design (at the very least, in the debrief).
Here’s an example (I’ve use this one often). This is the Microsoft Word font dialog from Word 2010. If my memory serves me correctly, this version may be less complex than the previous font dialog.
Note: I’m choosing this dialog because it’s interesting. I don’t know who tests this, and I bet they will have much better ideas on how to test this than me, but that doesn’t matter.
Ignoring the advanced tab and the functionality of any of the buttons other than OK or Cancel, how would you test this?
You can spend as much time as you want (or use as little time as you need). I don’t think I care about the actual tests as I care about how you decide what to test.
As always, you don’t need to actually reply to this post – just do the exercise if you want. I’ll write up some thoughts in a week or so and see if they make sense (I don’t think I have any idea where I’m going with this.
Hi, Alan…
“Participants in the game are expected to come up with functional tests (can it staple), as well as ‘out of the box’ tests like, ‘what happens if I drop the stapler off the top of my house’ (I guess it’s a form of a reliability test).”
I appreciate your use of the word “sometimes” in the sentence that precedes this one, but it’s also important to think about agency. “Are expected” begs the question of “by whom?” When I give someone a testing challenge like this, the functional tests might be interesting, but I usually want to see (first) the thought process involved in deciding the scope of the assignment, the gathering of context information, the development of oracles, and some ideas about what to cover and how to cover. I’m usually alarmed to see functional tests come out too soon, since that’s a common testing pathology. If I’m in a teaching mode, I’ll immediately respond to too-early functional tests by noting that I intend to use the stapler as a paperweight or a doorstop or the anchor for a helium balloon.
Adam Goucher once quoted me as saying that everything is a trap. That was a misquote. Everything is potentially a trap, and so one of the important and early tasks for a tester is to look for potential traps and get around them.
These are fun games, and while I suppose they do force some creativity, I don’t think they really have anything to do with testing software.
Naturally, it depends on the approach to the exercise, but I believe it has a ton to do with testing software, precisely because I see so many people rushing into the test execution part (and, heaven help us, the automated execution part) before understanding the context and the mission. There are plenty of candidates for Worst Testing Pathology, but premature design and execution is surely one of them.
That’s an important distinction for me – the goal of testing isn’t to find bugs. Finding bugs is merely a side-effect of the testing process.
I’d suggest some caution and precision here. For many organizations and many testing clients, their goal IS to find bugs–important problems that threaten the value of the product. To such clients, bug reports are not a side effect, but the desired deliverable, and those clients could be baffled by what you’re saying, or by the way you’re saying it.
I’d suggest a slightly different tack: I’d say that the quality and relevance of our reporting is set by the relationship between ourselves, our clients, the product, and the mission. So, not to give away the farm here, I observe the following things:
– I presume that you are my only client. I ask you to correct me if I’m wrong about that. I also presume that you don’t mind me asking questions and making observations, and that you’ll respond to those questions, or correct those observations when I err.
– I don’t have a copy of Word 2010 here, so I can’t be sure that this is really a font dialog from Word 2010.
– The font dialog from Word 2003 (which I do have) appears substantially the same, but with additional checkbox options Shadow, Outline, Emboss, and Engrave. Presumably those still exist under the “Text Effects” button, perhaps with even more options.
– A dialog box is has its own behaviours and state, but it is also closely related to many other functional and parafunctional aspects of the product. The mission here could be to test the box to see if its own functionality has some observable effect within itself; it could be to examine how changes to settings within the dialog produce an effect on an insertion point or some selected text; or it could be to explore the world of fonts and all of their interactions with document, images, graphics, effects, page layouts and the like.
– From those observations and my knowledge of the fact that Microsoft shipped Word 2010 some time ago and Word 2003 some time before that, I presume that someone has already tested this dialog, and presumably you want to revisit that testing, to extend their work, or both.
– You say “you can spend as much time as you want (or use as little time as you need).” Yet you’ve also noted that your goal is to model real testing assignments. In the real world, the schedule is rarely irrelevant; it’s usually one of the dominant context drivers.
So what’s my mission here?
I don’t think I have any idea where I’m going with this.
No problem. We call this exploratory research. 🙂
—Michael B.
“the goal of testing isn’t to find bugs.”
Can you clarify this? What is the goal of testing, then?
To provide information, to highlight risks, to critically examine the software, etc. Finding bugs is a side-effect – not a prime directive.
It seems like all those things you describe really boil down to bugs, though, no? The information you provide: what and how many bugs were found. Highlighting risks: looking at the bugs, grouping them into related areas to assess relative risk.
Not finding any is, of course, still valuable information. Not finding any doesn’t mean you aren’t a good tester, necessarily. But what else are you looking for, if not bugs?
You’re looking for risk – sometimes that manifests in bugs.
Perhaps it’s just semantics, but I find that changing my approach to evaluation and examination provides more value than looking for bugs.