I answered a few questions this week about automating GUI tests. One question was about recommendations for GUI automation tools for non-coders, and the other was about how much time to spend on the GUI in an MVC (model-view-controller) application.
The answers were easy. In the first case, I said that they weren’t going to get ROI from the effort, and they should just test the GUI manually. In the second case, I suggested that they do all of the automation ignoring the view/GUI, and test the GUI manually.
I could expand an entire post on why I gave those answers, but it doesn’t matter. I’m going to go out on a limb and make the following statement.
For 95% of all software applications, automating the GUI is a waste of time.
For the record, I typed 99% above first, then chickened out. I may change my mind again. The point is that I think testers, in general, spend too much time trying to automate GUIs. I’m not against automation – just write automation starting at a level below(*) the GUI and move down from there. I’m not saying that you shouldn’t test the GUI at all – I just don’t see why you wouldn’t want to test it manually, and get people knowledgeable in user experience to help. I just think that in most cases we are wasting our time when we try to automate GUIs, and wonder if anyone has the guts to stop.
* What I beam by a “level below the GUI” is automation that works with IAccessible or an object model rather than interacting with UI elements directly.
YES!!! I tried this (GUI automation) with enthusiasm twice and both times it was too expensive to maintain and nearly impossible to get rid of the false errors. Automation which costs more to create, run, and maintain than the same test run manually is a bad investment and it isn’t recouped if the tests have to be maintained and rewritten ever version. This doesn’t even account for the terrible job of validation that GUI test tools are doing.
In practice I say 99% although in theory I could agree with 95% with the hope that better ways to validate are possibly on the way along with better tools? Snapshots, screen captures, and “pixel diffing” sucks so bad. We need tools that validate WAY better.
I don’t lump object model and accessibility interfaces together. I see object model as below the GUI. Accessibility interfaces are testing through the GUI – there is a one to one correspondence with elements in the user interface. For disabled people, it IS the user interface.
Yep – you’re correct. Readers take note.
But how about test automation for stability purposes. Say I have a scenario where I test a set of functions 10000 times in order to find meomry leaks, file system issues or UI freezes?
Wouldn’t scripted test cases suffice? I mean the ROI is there.
Of course you could argue that simulating the same actions on low level APIs would do the work faster. But again, that would not be the same as a user would interact with the system.
Oh, just realized this blogpost is really old…
My next post will address these comments (they’re all valid concerns)