I frequently go days at work without writing code. On many days, I do “typical” test activities (seeing what works, what doesn’t work, etc.) Some days I’m debugging all day. Other days, I’m sifting through the perl and batch scripts that bolt our build process together. On other days, I’m managing projects, tweaking excel, or trying not to cause too many problems in SQL. This afternoon, at least after lunch, was coding time – and probably the first extended code time I’ve had in a few weeks.
Without too many boring details, I had a backlog item to write a service to automatically launch a memory analysis tool and periodically take snapshots of memory allocations in specified processes. The memory analysis tools (which I also wrote adapted from another project) work really well, but need a lot of manual intervention in order to get accurate data. I wrote a quick “spec” on my whiteboard, wrote tests as I went along, and in about four hours I had it up and running successfully. I’m confident that the service will make it much easier for our team to find and isolate memory issues. To me, writing this tool was a testing activity. It doesn’t test anything, but certainly makes testing of the product much easier.
On my drive home, I was pondering the universe (and my day) and remembered that there’s a sad bit of truth in the software world. It’s a fallacy that causes great confusion. It promotes poor strategies, bad hiring decisions, and endless confusion among management, contributors, and consultants. It may very well be the most misunderstood word in the world of software. It’s the “A” word that is practically ruining testing.
Automation.
Yes – as much as I hate them, I used a one word paragraph. I did it because the abuse and misuse of “automation” in testing is causing too many smart people to do too many dumb things. That said, I suppose an explanation is in order.
Once upon a time, someone noticed that testers who knew how to code (or coders who knew how to test) were doing some wicked-cool testing. They had tools and stress tests and monitoring and analysis and they found great bugs that nobody else could find – “Test Developers” were all the rage…so we (the industry) hired more of them. Not much later (and possibly simultaneously), someone else decided that testers who could code could automate all of the tasks that the silly testers who didn’t code used to do. Instead of finding really cool bugs, coding testers were now automating a bunch of steps previously done manually. Not to be left out, UI automation frameworks began to sprout, with a promise that “anyone can automate – even a tester.” Now, test teams can spend the bulk of their time maintaining and debugging tests that never should have been automated in the first place.
And that was the first step into darkness.
This mindset, although common, kills me:
- Testers who code write automation
- Automation is primarily made up of a suite of user tasks
- Automation is primarily done through the GUI
All of these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test teams in the wrong place and is, frankly, insulting to testers who know coding.
If you’re a tester who writes code (or a coder who cares about testing), try this:
- Write tools that help you find important bugs quickly
- Write a few high level tests that ensure user actions “work”, but leave most of it for exploratory testing, user testing (assuming you have great monitoring tools), or both.
- If you must write GUI tests for the above, write fewer of them (that way you’ll spend less time investigating their flaky behavior).
If you’re a tester who writes code, you’re not a test automator, you’re not a test automation engineer, and you don’t write test automation or test code.
You write code. And you test.
You write code that helps the product get out the door sooner. And if the code you write (or the testing you do) doesn’t help the product, why are you doing it in the first place?
I love/hate this post…
Love it because it has the loud sound of a nail being hit right on the head
Hate it because you’ve had to write such a post because of the all the crap surrounding the testers should code/automation cures all ills crap
Bookmarked in my favorites
You should replace your keyboard, it’s the second article with a missing closing Parentheses):
“If you’re a tester who writes code (or a coder who cares about testing, try this:”
It’s the damn script compiler…(or bad editor) :}
Basically I agree with your “what not to do”, but I am not sure what are your recommendations- leave everything to manual testing (exploratory testing and user testing)? while this is good in terms of pure quality, it is not very efficient if you have a lot to release and short schedules.
Alan,
As a tester who writes “test automation” code I can agree with you regarding your main gripe. Again testing is in this “automation frenzy” phase, just like it has a couple of times before, and people & companies are getting themselves into deep do do.
It’s what I like to call the “automagic” problem. Just automate your testing and all your problems will be solved. Uh… not quite.
It springs from my favorite saying of “It’s Automation, Not Automagic”. Too many people/companies are diving in head first with trying to implement automation with a tool and not understanding first why they want to automate, what to automate, and who best to build it. Also, there are some vendors, again, out there touting their “solutions” as the way to go. Caveat Emptor!
Just amazing… we keep proving Einstein right about Insanity. And it looks like from your post that you are going a bit crazy right now. Time for some heavy drinking.
Look at it this way, all of this keeps us employed. 🙂
Best to you.
Jim
Unfortunately, you might be on to something here.
A lot of “test development” is really just GUI automation, which you correctly point out can be worse than useless. It gets even worse when this test code isn’t really considered “code” so it just thrown together and then held up using the coding equivalent of chewing gum and a prayer. A “manual” test that could take a minute to perform all of a sudden takes multiple employees weeks of time to run and maintain in a script.
That being said, there’s definitely a place for automated testing, even UI automation. Tools are getting better and some folks are catching on to this idea of “test code is code”. When done well, these automated tests can be run continually, not necessarily to find new bugs but to shorten information feedback loops and to raise lower limits on quality.
Think of automated UI tests as the new unit tests.
I’d like to call attention to this:
automate all of the tasks that the silly testers who didn’t code used to do
That implies that there was heavily scripted manual testing. I don’t mean script in the sense of code. I mean a script like an actor reads. I mean all of those very verbose test cases with all of the very explicit steps from which one should not deviate. Reread that last sentence aloud with some sarcasm in your voice if you found yourself arguing with it.
Automating a bunch of crappy robotic test cases only makes sense if you have a bunch of crappy robotic test cases that you need to slog through manually over and over. Because fast crap is better than slow crap. I have no problem with that at all except that it’s crap.
I think the ire directed at automation, particularly GUI automation, is misdirected. Problematic GUI automation may be an indication that the GUI is a poor API but it may also mean that the original manual testing that is being replaced had problems that nobody was paying attention to. Manual tests that treat humans as programmable agents are strange, misguided attempts to turn smart, thoughtful work into something done on an assembly line. In my own experience decent manual testing (exploratory methods et al.) simply can not be replaced with automation.
All that said, I code and I can’t imagine why any software tester wouldn’t want to. And even the “manual” testing I do is augmented with tools.