I’ve been pondering test automation recently. Maybe it’s because of my participation in the upcoming stp summit (note: shameless self-promotion), but that’s only the surface of it. I’ve complained about misguided test automation efforts before, but it’s more than that too. For every tester that cries out that 100% automation is the only way to test software, someone else is simultaneously stating that only a human (with eyes and a brain) can adequately test software.
The answer, of course, is in the middle.
But I worry that even for those who have figured out that this isn’t an all or nothing proposition, many testers have no idea at all how to design an automated test – which means that they don’t know how to design a test in the first place. The problem I see most often is in the separation of automated and human testing. When approaching a testing problem, you have failed if your first approach is to think about how you’re going to (or not going to) automate. The first step – and most important – is to think how you’re going to test. From that test design effort, you can deduce what aspects of testing could be accomplished more efficiently with automation (and without).
A common variation of this is the automation of manual tests. It pains me to hear that some testers design scripted test cases, and then automate those actions. This tells me two things: the manual test cases suck, and the automation (probably) sucks. A good human brain-engaged test case never makes a good automated test case, and automating a scripted scenario is rarely a good automated test (although occasionally, it’s a start). Some teams even separate the test “writers” from the “automators” – which, to me, is a perfect recipe for crap automation.
An example would be good here. Imagine an application with the following requirements / attributes:
- The application has a “Roll” button that, when executed, generates five instances of random numbers between 0-9 (inclusive)
- The application totals the output from the 5 random numbers and displays them in the “Total” field.
- There are no user editable fields in the application
For those of you with no imagination, this is how I imagine it.
From a manual only perspective, layout, user experience, and interaction are definitely areas that need to be investigated, and that are usually best done manually. If I were to write a few scripted manual test cases for this (not that I would), they may look something like this:
Test Case 1
- Press Roll button
- Count (use calculator or an abacas if necessary) to verify that the value in the Total field matches the sum of the values below
Test Case 2
- Press Roll
- Ensure that the values in the lower fields are within 0-9 inclusively
- Repeat at least n times
Test Case 3
- Press Roll
- Ensure that the value in the top section is between 0 and 45
- Repeat
I have two complaints about the above tests. The first is that executing them manually is about as exciting as watching a banana rot, and the second is that (as I predicted), they’re not very good automated tests either.
When designing tests, it’s important to think about how automation can (or won’t) make the testing more efficient. What I hope you’ve realized already (and if not, please take a moment to think about the huge testing problem we haven’t talked about yet with this application), is that we’ve done nothing to test for randomness or distribution.
Testing randomness is fun because it’s harder than most people bother to think about. We don’t have to get it perfect for this example, but let’s at least think about it. In addition to the above test cases (granted, the third test case above may be redundant with the first), we need to think about distribution of values within the bottom five boxes. Given the functional goal we’re shooting for, we can probably hit all those test cases and more in a reasonably designed automated test.
Pseudo-code:
Loop 100,000 (or more) iterations
Press Roll Button
Verify that sum of output and total field are identical
Verify that output values are between 0 & 9
Store count of values from output boxes 1-5
End Loop
Examine distribution of numbers
Examine sequence of numbers
Other pattern matching activities as needed
The first loop takes care of the main functionality testing, but where automation really helps here is in the analysis of the output. I’d expect a statistically accurate representation of values, repeated values, and sequences.
That’s stuff you want to use automation to solve. Yet, I keep discovering stories of testers who either don’t bother to test stuff like that at all, or put their automation effort into trying to navigate the subtleties of UI testing (yet something else I have an opinion about).
I don’t’ care whether you’re an automator, a tester, a coder, or a cuttlefish – your goal in testing is not to automate everything, nor is it to validate the user experience through brain-engaged analysis. Your job is to use the most appropriate set of tools and test ideas to carry out your testing mission. You can’t do that when you decide how you’re going to test before you start thinking about what you’re going to test.
Notes: I don’t know why I picked 100k for the loop – it may be overkill, but it’s also something you have an option of doing if you’re automating the test. I suppose you could do that manually, but you will go insane…or make a mistake…or both.
There’s more to this concept, and follow ups…maybe. The big point is that I worry that people think that coded tests replace human tests, when they really enhance human testing – and also that human testers fail to see where coded tests can help them improve their testing. I think this is a huge problem – and that it’s one of those problems that sounds bad, but is actually much, much worse than that…but I don’t know how to get that point across.
I should also point out that given my loathing of GUI automation, that I’d really, really, hope that the pseudo-code I scratched out above could be accomplished by manipulating a model or an object model or the underlying logic directly. I want my logic/functional tests to test the logic and functionality of the application – not its flaky UI.
I think it’s worth asking why you’re testing what you are through the UI at all. And whether you couldn’t maybe hit interesting boundary conditions if you controlled the PRNG, maybe even replacing it with a CDNG (“completely deterministic number generator” – I just made that up).
I’m not trying to be contrary. I’m actually reasserting your claim that the automation is bad because the test design is bad. And I’m asserting the claim that testing the way internal data is handled via the UI is wrong – something I know you agree with, based on what you’ve written before. Or were you going to go there in a future blog post anyway? Sorry if I stepped on your line . . .
I never know where I’m going.
The main point I was thinking of when writing this was that too many testers tend to dive right into testing without thinking about testing.
“There’s more to this concept, and follow ups…maybe. The big point is that I worry that people think that coded tests replace human tests, when they really enhance human testing – and also that human testers fail to see where coded tests can help them improve their testing. I think this is a huge problem – and that it’s one of those problems that sounds bad, but is actually much, much worse than that…but I don’t know how to get that point across.”
Agreed on all counts!
I think it’s always important that good testers have lots of tools in their bag of tricks. Some of those “tools” are actual tools such as test automation, and some of those “tools” are techniques that can be applied in appropriate contexts.
I think this is part of why I dislike having automation teams distinct from manual test teams. Test automation versus manual testing is not “either/or” – it’s almost always “and”.
Glad to have found your page, Page!
I totally agree what you say about testers refusing test automation to help them reach better testing.
Testers at my company acutally believed that they were to get the sack if we introduced test automation tools…
I will start to follow your blog closely from now on as fools seldom differ… 😉
Alan,
Great post, and I agree on most of what you say. My only contention is regarding GUI level automation. I think a lot of time that is the layer you’re stuck working with. I know because that is what my clients give me, getting developers to open up that lower layer (API or lower) is tough to do with an existing system. It has to be ‘baked in’ so to speak from the beginning. Now TDD is helping to drive that, but there is still a lot of legacy (both Fat client and now Web based) systems out there that don’t have the ‘hooks’ in place to utilize.
The key here is to design your automation & tests to go to that lower layer if you can while also working through the GUI. The problem I see in addition to what you talk about (preliminary work and planning) is that there are too many ‘automators’ who don’t have the programming/technical skills to beyond the basic Record/Playback & strict GUI layer methods. That is a failing of the tester themself to not know more about the system they are working with or the tools capabilities to go beyond the GUI layer.
I’m glad to see things such as TDD gaining traction, the use of Unit Test harnesses and middle layer tools. But these skills seem to be predominately with the developers, more testers need to pick up these technical skills.
And yes, automation is only a tool. It is the person and the computer between their ears that make proper use of the tool.
Automation should be used for an efficiency gain via distributed testing (spread the workload) and eventual parallel execution. Machines are cheap, and as you have shown/talked about before you can do it even better and cheaper via virtualization.
I’m starting to ramble (which Joe knows I’m notorious for), but I think we agree on 99.9% of things. Bottom line is this: Automation is a tool only and as such needs to be used intelligently. It is not a Silver Bullet and isn’t magical (It’s Automation, Not Automagic). It needs to be planned, implemented and maintained like any other piece of software. But it is only as good as the humans who create it.
I planned to talk about this in my next post – stay tuned (and thanks for reading and commenting)
Hi Alan, great post. My question is… and what about regression testing? I mean, if you just automate the code layer, should you not have at least one test that tests the ‘Roll’ buton from the interface? Knowing this button has stopped working would be nice, and knowing it before the customer would be nicer too!
If all of your testing is automated, this would be a good regression case to add. Most of the time, I like to at least take a look at the application with my eyes before shipping it to a customer.
The “eye-time” also gives me a chance to make sure the background color didn’t change to pink.
… didn’t change to pink, damn yeah.
I’ve read a lot about this subject, and still, I find myself making the same question, I suppose that there is a certain path you follow as you grow as a tester, something like when you understand A you start understanding about B. Maybe this is why testers keep talking about the same subjects as years go by.
Somehow, we testers got this job when we where working elsewhere before, but also, many quit before they reach some level of knowledge (or stuck at some level of ignorance) and continue delivering work at some other job. (if having knowledge on other disciplines is good to be a tester, you might think being a tester can be usefull to work in many other jobs).
And maybe we could trace a path for bad automation, first you try to automate everything, then to automate GUI, then to automate manual tasks, then to automate test management…
and then, you start undestanding ..
Here it is:
http://cartoontester.blogspot.com/2010/05/career-planning-for-sw-testing.html
But it might also work the other way.
I think it’s interesting that none of the commenters so far has raised this question (since the whole issue here is how you would test and all): What IS “a statistically accurate representation of values, repeated values, and sequences”?
—Michael B.