I’ve been gathering some information on test automation systems recently and will probably have a flurry of posts upcoming on automation and related subjects.
Some observations as I browse what Bing has to tell me about the subject of Test Automation:
- Wikipedia tells me that test automation “commonly involves automating a manual process already in place”
- The bulk of the test automation articles are about UI automation
- There are some reasonably good articles warning about the problems one may run into in test automation projects. None of these articles provide solutions or alternatives.
There’s more in my search results, but those three results alone say a lot about what’s wrong with the way (most?) teams approach test automation.
I expect I’ll write more eventually on all of these points, but my anecdotal experience, along with a few dozen web pages and articles tells me that there’s a lot of confusion in regards to test automation.
You see, test design (including the design of how test automation will execute) is really, really hard. It’s hard to write sustainable and reliable tests, and it’s hard to write trustworthy tests. Double the effort required if those tests involve UI automation. Designing good tests is one of the hardest tasks in software development. That’s worth saying again. Designing good tests is one of the hardest tasks in software development.
Compared to everything else in the equation, the “writing code” part of test automation is easy. Massively easy. Almost brain-dead easy. When I compare writing code to composing music, I talk about the creative and challenging aspect of melody, creating a texture and mood, and figuring out where notes and space between notes help with both. There’s also the rote part of the job – chord voicings, doubling, and other parts of filling in the score. The melody and texture choices are the test design of music composition – filling in the rest of the parts is the coding part. Sure, there’s creativity in a countermelody, as much as there’s creativity in a cool algorithm, but it’s still the rote part of the activity.
I fear that testers are worrying too much about the effort and skills required to automate tests – and not enough about what it takes to design reliable, portable, accurate, and trustworthy tests that actually matter.
Tell me I’m wrong.
It’s really really hard to do in isolation, but it’s merely difficult when collaborating with the business and the programmer (what I call the “three amigos”). It still seems hard for people to talk about the “what” instead of the “how.”
I wish I could tell you were wrong – but a quick perusal of QA Stack Exchange shows it’s all questions of how to access a web element or weddriver not working, questions on STC are nearly all on ‘what is the best tool for automation’ , no questions about how to design the tests.
Looking forward to the rest of your posts on this
You are not wrong.
I am mentoring a growing automation effort (in Python) and indeed one of the greatest challenges is the design.
The developer can hack a test in a day, but he will choke when he’ll have to add the other 10,000 variations for it, analyze results and create smart reports.
I always make my automation developers approach their projects the same way as any other software project- start with a high level design, then the details (and code review it, follow coding guidelines, version control it, unit test it etc.)
Alan,
Where to start… you are correct in your statements and observations. More so than most people will want to admit.
The ‘design’ of a test is the key thing; both manual and automated. The other thing to realize is that ‘automation’ in the typical sense, especially with front-end/UI, is that it is automating the ‘execution’ of the test only. The design, construction, analysis and other tasks that are part of a human doing testing isn’t considered by most regarding automation.
A big thing as you point out are the misconceptions/misunderstanding of what all is involved in ‘automating’ test execution. Too many times people & companies don’t see this type of work for what it is; software development. We need to think about how we build and maintain the ‘testware’, and who is best to do this type of work.
This comes from the problems of the “Automagic” wishful desires and salesjob by vendors. Management doesn’t want to hear that automation is another form of software development. Testers are not Developers, thus you don’t need people (and pay for them accordingly) with those skills and mentality.
You know the story… being at Microsoft you have been through this. But a lot of other companies have not and/or don’t want to due to various reasons (1 being cost and the other ignorance).
I’d love to rant on more, but this is a beer/phone conversation and my fingers are tired enough as is.
Looking forward to other posts you have on this subject. Just remember my favorite saying/axiom: “It’s Automation, Not Automagic!”.
Regards,
Jim Hazen
Alan,
This is so very true. Designing the tests is the critical aspect, and I’ve suffered from far too many managers who don’t get that the combination of skills to both design tests and convince software to perform them is vanishingly rare. (It can be learned – but the people who want to learn aren’t that common either).
From my perspective, automated regression is what assures customers that the critical parts of the software meet regulatory requirements – since my employer supplies commercial software to large businesses, that means compliance with various tax rules, privacy rules, payment data storage rules, and so forth.
Alan,
great to see you at SASQAG last week!
I think part of the problem is that Test is much less mature as a discipline than Dev is. This follows from Test being lower priority, and less prestigious. Of course, that works for developing games, but for enterprise software you really need to get serious about quality, and test automation is an important part of that.
I’d love to hear more of your thoughts on this topic. I’d love it if you added your comments to a blog post of mine that you find interesting! Like you, I’d like to learn from the experts.