I’ve been having one of those experiences where a lot of the random bits of flotsam on my radar are sort of converging. But I don’t know if it’s really convergence, or if I’m forcing it. I hope it’s the former, but time will tell.
In my last post, I reflected on a wonderful article from Phillip Armour. His breakdown of “tests that tell us what we know” (0OI tests), and tests that tell us what we don’t know we don’t know (2OI tests) aligns perfectly with something that’s been gnawing at me for a long time.
You see, I’m also bothered (and have been for years) to some extent by one of the more popular testing debates – the scripted vs. exploratory tests…and to top it off, I’m completely confused by the variety of reactions to the “test is dead” statement one of my old friends has been spouting about recently (I warned you about random thoughts floating in my head…)
But it’s kind of coming together for me – some of it as I write these paragraphs.
A part of the problem is that what I call testing is probably different than what you call testing. That’s also true if I turn it around – what you call testing is probably different than what I call testing. The difference is, that I don’t care what you call testing, and some of you
care a lot get really pissed off when somebody calls something testing that doesn’t match your definition.
Testing, in some intersection of our combined testing definitions, contains a lot of 0OI tests. “When you enter 2 + 2 in the calculator, the display shall enter 4”, or “Save the file and ensure it is created”, or “When the user clicks the submit button without a user name, they shall receive an error message”. Many in the test community refer to these as “checks”, and I’m ok with that term, but to me, these are a type of test rather than something different than tests.
Whether you call these tests, checks, or 0OI tests (my current favorite), I don’t want to do them. It’s not just that 0OI tests bore me (because they bore me to tears), but because I think it’s fiscally irresponsible to push verification of mind-numbingly basic functionality so far downstream. In my opinion, developers should do this work, but I guess if any testers find 0OI tests entertaining, I’m sure there’s a lazy developer somewhere who’d like to hire you as his cabin boy.
Short story is that 0OI tests are dead to me.
And then there’s 2OI work – the kind of work good testers love to do. 2OI testing is the land of “I wonder what happens when…”, and “How can I get the program to…” tests. 2OI tests can be manual or leverage tools or automation. Any form of discovery (or new knowledge acquisition) about the software is 2OI testing.
Testers not infatuated with Armour the way I am typically call 2OI testing exploratory testing (true for both manual or automated 2OI tests). In Armour’s recent article on testing that I mentioned in my previous post, he suggests that the sweet spot for 2OI (exploratory) effectiveness is about 50% – meaning that about half of your tests should discover product issues. If you’re finding less, you could vary your techniques, and if you’re finding more, you should back off the effort. I haven’t applied that concept yet when performing testing, but I plan to the next time I conduct a testing session.
But, the way I see it, exploratory testing is dead too. OK – that statement was overly controversial. What’s dead to me, is the term exploratory testing. To me, it’s just testing. It’s all just testing. I wrote a blog post once about this subject (see last three paragraphs to avoid most of my snarkiness) – then one of the top practitioners of ET wrote this post with a similar stance – but it seems every tester in the world is still compelled to proclaim they are “exploratory testers”, because they like to use their brain while they’re testing. All of this is fine, of course, one can use whatever terms they like, but to me, at least, it’s all just testing.
Testers do some of it. Others do some more of it. Some of it’s dead to me, and perhaps none of it is dead to you. It doesn’t matter.
It’s all just testing.
I don’t like the term “It’s all just testing” as I have heard it used as an argument for not having to analyze what kind of testing you are doing. Maybe we could call using brains while testing Real Testing(tm)
For me exploratory testing is a term for certain type of testing where tests are based on broader ideas instead of more detailed steps and tester guides the execution. Term is a useful shortcut.
I wrote while a go why exploratory testing to me feels better than tests written beforehand. Short summary is that detailed steps and expected results give tester a bar for acceptable and do not encourage testers to challenge the tests even if they are thinking while testing. In exploratory testing the tester has to decide when testing is done.
Of course it is possible to do things well or badly using nearly any system. The interesting question is which ways guide testers towards better testing.
Reading your text again it seems that I kind of missed your point of defining again the meaning of testing (as you use the word).Anyway, it still helps to have categories to help communication with people involved. Some vategories not being inherently better than others, but each done if suitable and in a sensible way.
Good article, Alan. Agreed – it’s all just testing.
I think what’s behind the increased splintering of the word “testing” into related terms is the desire for some folks to feel like what they are doing isn’t “just testing”, it’s more important than that, better, superior.
If I’m doing “exploratory testing” that must be better than simple testing right? And if you are doing checking, that must be worse than testing, right?
If you are only doing “GUI testing”, that can’t be as important as my “API testing”, can it? And if I’m doing “performance testing”, well that’s just something special, certainly worthy of far more praise than mere “functional testing”!
I’m hoping people are doing “whatever kind of testing is appropriate for your company at this time”. And to keep the conversation shorter, I’ll just call that “testing”.
I agree with you and Joe. In the end it is all “testing”. The need to split hairs on terminology and meaning is just plain dumb and worthless in my book. It seems for some people it is their way to differentiate themselves and somehow make themselves better than the average person. This for some consultants/pundits is their way to justify their views (and rates). Which I don’t care about, some of what they have done has advanced our profession. But in some situations it is now starting to hurt in my opinion.
I have a B.S. in Zoology and as part of my studies I did a lot of labs & experiments (I ‘tested’ theories). I’ve translated this into my work in software testing. To me the bottom line is that a ‘test’ is a way of proving or disproving a theory. This could be a ‘check’ (validation) as some call it to revalidate a proven theory (it worked before, it better work again). Or it could be to ‘explore’ (experiment with something new) to see how it behaves (according to your newly thought of theory). This will prove or disprove your theory.
So in the end either being a check or exploring you are doing some type of experiment (or test) of the software. You are either proving or disproving your theory (perception or expectation) of the software.
Thus, in the end it is all ‘testing’ in my book too.
I am confident we need more words about testing, more categories and stories that help us understand, explain, and advance our profession.
But we don’t need context-free values about which ways are better, we don’t need to put people in boxes, and say some are more worth.
Rather say “In that situation, I would have preferred to try this and that”.
We should also be careful to assume why people create and use categories.
For example, my interpretation of testing/checking is that both are part of testing, and that the distinction is very powerful when explaining different kinds of testing efforts, e.g. what developers can accomplish with unit tests, and why manual testing is needed.
My experience is that the words make this easier, they stick.
I would also avoid saying it’s JUST testing, as it is easy to believe it is simple.
I like the way you’ve included Phillip Armour’s OI to types of tests. You’ve helped me to understand why certain tests niggle me – these tend to be 0OI and the sort of thing I’d want to automate, if they’re worth executing at all.
One of the key testing skills for me is when we focus on 2OI and are competent and addressing 3OI.
Cool – I was hoping that would make sense (and that the 0OI tests would niggle someone besides me).