Despite the lame title, this isn’t yet another post attempting to describe what testing is – it’s just something I wanted to share inspired by a small discussion last night (although if you browse the testing blogosphere, you’ll find a lot of good discussion on this topic lately).
I went to Seth Eliot’s talk at SASQAG last night. Afterwards, Seth, Keith Stobie and I were talking (actually, Seth and Keith were talking, and I was listening), and Keith mentioned that some of what Seth talked about wasn’t actually testing. Seth said, "sure it was.", but Keith disagreed. Finally Keith said, "not every bug finding activity is testing." An example was that Seth was talking about analyzing customer log files in order to find bugs, and calling it testing. I suppose technically, that’s an analysis activity and not a testing activity…or is it? Another example are code reviews – they’re a valuable bug finding activity, but certainly not a test activity…
I think it’s easy to blur the line between what testing is, and what testers do. I’m not convinced it’s correct to blur the line (or incorrect for that matter), but I do think it’s a frequent cause of confusion among testers. If you were to ask a tester on my team, for example, what they do, they would say "I’m a tester". But if you asked them to elaborate, they’d say something like, "I review code and product designs, I analyze customer data, I write automated tests, I debug and analyze test failures, I identify risks (and investigate risk mitigation), I measure things, I explore the application, work on cross-group initiatives, I work with developers, I debug, I try out new ideas, I work on quality initiatives, etc.).
I suppose you could look at that in two different ways. One is, "wow, the testers on your team do a lot of stuff," or "wow, the testers on your team don’t do very much testing". Either answer is fine, as the answer depends entirely on your definition of testing, and whether you think testers should only do testing, or whether they perform analysis and prevention tasks as well.
The question is, how much of this is testing? I think I’ve always considered all of these things to be part of the testing world, but I know others see testing as only one or two of those activities. I’m not saying anyone is right or wrong in coming up with their definitions of testing; instead I realize even more than ever that the different views can cause communication problems (as of right now, I’m sort of leaning toward calling very little of this stuff testing, but I may change my mind tomorrow). I think the view differences are largely contextual – on many teams testers do just a subset of that list, and that’s perfectly fine. I suppose that’s just another reason why titles mean little, and experiences and activities mean much, much more.
I was going to ask for reader comments on what testing was, but I don’t think consensus (or verification that there are different views) is important here. What I think is important is that every tester and test team define the role of testing for their context, and do what’s feasible and practical given the product, team, schedule and other factors.
What I think is important is that every tester and test team define the role of testing for their context, and do what’s feasible and practical given the product, team, schedule and other factors.
In a community of like-minded people, you might see a number of defintions.
James Bach says that testing is “Questioning a product in order to evaluate it.”
Jerry Weinberg says that testing is “gathering information with the intention of informing a decision.”
Cem Kaner says that testing is (deep breath, now) “A technical, empirical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek.”
I say that “testing is the investigation of systems composed of people, computer programs, and related products and services.” What’s more, I agree with the three above, since, broadly speaking, we all mean more or less the same thing.
While I think it might be a good thing for Keith to look up “assimilation bias”, the question that we might want to ask is this: how might a particular definition interact with our behaviour? Would it make a difference?
I’ll suggest one way in which it might make a difference: in my practice, and in my observation of others’ practices, narrow definitions tend to increase the chance that we’ll fail to recognize something important. For example, Keith seems to regard testing as test design and test execution, yet (as you point out) these are not the only valuable things that testers and testing can do. Notice, though, how the label creeps from the activity of testing (which is apparently what Keith and Seth were talking about; what testing is) into the role of testing (which is what you’re talking about now; what testers do).
I would argue that such slippages can be dangerous, leading to misinterpretation or narrowness of thinking. Even if you, Alan, believe that a Software Development Engineer in Test doesn’t necessarily mean “programmer”, the perception for others—including some hiring managers at Microsoft, and some applicants for the job—will surely influence the kind of people who will apply (or don’t), and the kind of people who will be hired (or won’t).
So: I agree with you. Every team needs to invent and describe and understsanding testing for itself. And I would argue that the same goes for every tester, too. Yet I would also suggest strongly an expansive view of what testing is, to reduce the chance that we’ll miss doing important activities and finding important problems.
—Michael B.
Thanks for taking time to comment Michael. I wrote the post because I found the blur between what testers do (or can do), and what people call testing vary widely. I completely agree comletely with your dangerous slippage comments as well.
Your comments on Microsoft roles are fair – I do not believe the SDET role is a programmer role, but I realize that I can’t speak for all of Microsoft. When I “sell” the job of testing at Microsoft to candidates during interviews, I talk extensively about the expansive role, and tell candidates that if they want a role that where programming is the primary focus that they should take an SDE job instead. While I’m probably not the only “senior”* interviewer with this tactic, I’ll bet it’s rare.
(*) – Sine I don’t manage people, I’m not a hiring manager in Microsoft’s terms, but I am an “As Appropriate” interviewer – meaning I’m the last interviewer candidates see and I make the final hiring decision.
Hi!
Interesting issue. Page shows that contains three coments, but there are only two.
This is not bug for me, maybe for you Alan, if you use this information for user activity on your blog.
WordPress shows trackbacks / pingbacks as comments. Fred Beringer linked to this post and I allowed the pingback request from his blog, so that “comment” shows up as just a link back to his blog post.
Interestingly, one of the slides from my recent talk was also in my previous talk (slide 47 at http://exp-platform.com/bsc2010.aspx). The slide is titled Testing in Production, and goes on to explain what I mean by “Testing” and “Production”. In the notes section (not displayed…the note only seen by the speaker) it says, “Define Testing? Not here, not now (I believe it’s about evaluating risk)” 🙂
On an unrelated note, I found myself Binging to find your blog. As you said your name is not unique enough, so I used “alan page weasel” and found what I wanted at slot #1 🙂
That’s definitely a good way to remember me – thanks for highlighting that, Seth.