I’m sure by now that most people who want to have seen the webinar I gave last week on test design (if not,the recording is here and slides are here).
I had a few goals in this presentation. One I stated clearly – this presentation was primarily about the how of Test Design – not the what. This means that I didn’t share a big list of test design ideas – in fact, I don’t think I shared any unless they helped make a point.
Other points I left (purposely) up to the imagination. I have an inkling that some testers think that there’s a master list of test design ideas somewhere, and once they have the list, they’ll be up for any testing challenge.
This, of course, is a horribly broken idea.
First of all, Test Design ideas (I’ve explored, and continue to explore the pattern metaphor with these ideas) are endless. Like infinity, as many testing ideas as you can list, I can think of one more.
The bigger challenge is knowing when to use which testing idea – and to me, that’s the bigger challenge in test design. Let’s say that I’m an expert in Boundary Value Analysis. You can stop laughing now – all that means is that I’m a hotshot at finding boundary issues. The problem is that only a small (but significant) percentage of testing actually deals with boundaries. I may try testing boundaries in a lot of places, but most of the time that effort isn’t doing anything for me.
As a tester, you need to recognize the problem your facing, then pick the best test design ideas for that problem.
Here’s a real example. It’s no secret that I like model-based testing. Sometimes, testers will ask me “how would I use model-based testing in this scenario?”. That’s a nice question, but it’s the wrong question. To be a good test designer, you need to look at the problem first (I need to test this application), analyze how the program works, then pick the best approaches. If you start with the approach and try to make it work, you’re probably going to fail. With this example, a tester with MBT in their “tester toolbox” may recognize a portion of the application that 1)behaves like a finite state machine, and 2) recognizes risk in traversing that state machine. Then they apply the technique.
The point is that to be a good test designer (and tester), you need a lot of testing ideas, and you need to know how and when to apply them.
And that’s really, really hard – and as you know, that’s also why it’s really, really fun.
Happy Testing
Hi Alan,
I do practice model-based approach in automating checks but I’m concerned that MBT is good enough for testing of real-time / cloud-based systems, and this is towards its all going.
In fact, as a long time reader of your blog, I recall examples from your series of the future of software testing.
“Putting the pieces together”
https://angryweasel.com/blog/?p=192
“My worry is that gigantic software systems are even more prone to critical errors due to an exponential number of code paths […]”
– That means, an exponential number of states to map and model, right? So it becomes pointless. Plus, the same functionality does not traverse exactly the same path on complex systems. I wrote about the latter in my blog post at
http://automation-beyond.com/2011/01/27/wta05-test-the-envisioned/ – “If not testers then who?” part.
Thanks,
Albert
To paraphrase what I was saying above, If MBT isn’t a good solution for the presented challenge, use something that is a good solution (replace MBT with any testing idea or approach.
In the case of the huge systems I described, a bit of prevention may be needed – e.g.
Step 1: minimize the dependenciesa
Step 2: test the crap out of them in isolation.
If there are too many dependencies to have time to test them all, redo Step 1 :}.
If, once the dependencies are known and understood, a state machine is still screaming at you, by all means model it – but you don’t necesarily have to model the entire system (my advice on model size: “too small is just right”). It’s ok to create several overlapping models, and certainly a preferrable solution if modelling the entire system is too complex.