{"id":332,"date":"2011-09-01T17:27:31","date_gmt":"2011-09-02T00:27:31","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=332"},"modified":"2011-09-01T17:27:31","modified_gmt":"2011-09-02T00:27:31","slug":"design-for-gui-automation","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/design-for-gui-automation\/","title":{"rendered":"Design for *GUI* Automation"},"content":{"rendered":"<p>I\u2019ve stated my distaste for GUI automation before, but after my last post, I feel the need to share some clarification. I\u2019ve been meaning to write this post ever since I wrote <a href=\"http:\/\/angryweasel.com\/blog\/?p=7\">GUI Shmooey<\/a>, so here goes.<\/p>\n<p>First off, let me state my main points for disliking GUI automation:<\/p>\n<ul>\n<li>It\u2019s (typically) fragile \u2013 tests tend to break \/ stop working \/ work unsuccessfully often<\/li>\n<li>It rarely lasts through multiple versions of a project (another aspect of fragility)<\/li>\n<li>It\u2019s freakin\u2019 <em>hard<\/em> to automate UI (and keep track of state, verify, etc.) <\/li>\n<li>Available tools are weak to moderate (this is arguable, depending on what you want to do with the tools).<\/li>\n<\/ul>\n<p>Let\u2019s ignore the last point for now. Partially because it\u2019s a debatable point, but more importantly because it\u2019s a potentially <em>solvable<\/em> point (one potential improvement would be if vendors would market these tools as test authoring tools rather than record and playback tools).<\/p>\n<p>The first two points above have more to do with test design than the authoring of the tests. It\u2019s possible to write robust (or at least less fragile) tests that work as the product evolves. Designing tests with abstractions and layers and other good design principles will help immensely. Doing this <em>right<\/em> drives the third point home. Designing robust GUI automation is difficult. Of course it can be done well, but I\u2019m not completely convinced it\u2019s worth the ROI. <\/p>\n<p>That\u2019s why it\u2019s important when designing <em>any <\/em>test to think about the entire test space and use automation where it helps you discover information about the application under test that would be impractical or expensive to test otherwise. I <strike>hate<\/strike> loathe automated tests that walk through a standard user scenario (click the button, check the result, click the button, check the next result, etc.) But I love GUI automation that can automatically explore variations of a GUI based task flow. The problem is that the latter solution requires design skills that many testers don\u2019t consider.<\/p>\n<p>Another example where I like GUI automation is in stress or performance issues. A manual test that presses a button a thousand times is out of the question, but it\u2019s perfect for an automated scenario. In fact, I wrote about something similar in <a href=\"http:\/\/www.hwtsam.com\">HWTSAM<\/a>. <\/p>\n<blockquote>\n<p><strong>Brute force UI automation       <br \/><\/strong>In most cases, UI automation that accesses controls through a model or similar methods tests just as well as automation that accesses the UI through button clicks and key presses. Occasionally, however, automating purely through the model can miss critical bugs.<\/p>\n<p>Several years ago, a spinoff of the Windows CE team was working on a project called the Windows Powered Smart Display. This device was a flat screen monitor that also functioned as a thin client for terminal services. When the monitor was undocked from the workstation, it would connect back to the workstation using a terminal server client.      <\/p>\n<p>The software on the device was primarily composed of core Windows CE components, but also contained a simple user interface that showed a history of connected servers, battery life, connection speed, and any local applications, as shown in the following graphic. The CE components were all well tested, and the small test team assigned to the device tested several user scenarios in addition to some manual functionality testing. Toward the end of the product cycle, I spent some time working with the team, helping them develop some additional tests. <\/p>\n<p><a href=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2011\/09\/image.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; margin: 6px 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px\" title=\"The missing screenshot\" border=\"0\" alt=\"The missing screenshot\" src=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2011\/09\/image_thumb.png\" width=\"442\" height=\"325\" \/><\/a>**<\/p>\n<p>One of the first things I wanted to do was create a few simple tests that I could run overnight to find any issues with the software that might not show up for days or weeks for a typical user. There was no object model for the application, and no other testability features, but given the simplicity of the application and the amount of time I had, I dove in to what I call brute force UI automation. I quickly wrote some code that could find each of the individual windows on the single screen that made up the application. I remember that I was going to look up the specific Windows message that this program used, but I\u2019d hit a roadblock waiting for access to the source code. I\u2019ve never been a fan of waiting around, so I decided to write brute-force code that would center the mouse over the position of the control on the screen and send a mouse click to that point on the screen. After a few minutes of debugging and testing, I had a simple application that would randomly connect to any available server, verify that the connection was successfully established, and then terminate the terminal server session.     <\/p>\n<p>I configured the application to loop infinitely, started it, and then let it run while I took care of a final few odds and ends before heading home for the day. I glanced over my shoulder once in a while, happy to see the application connecting and disconnecting every few seconds. However, just as I stood up to leave, I turned around to take one final look at my test application and I saw that it had crashed. I happened to be running under the debugger and noticed that the crash was caused by a memory leak, and the application was out of a particular Windows resource. At first, I thought it was a problem in my application, so I spent some time scanning the source code looking for any place where items had been using that type of resource or perhaps where I was misusing a Windows API.      <\/p>\n<p>I couldn\u2019t find anything wrong but still thought the problem must be mine, and that I might have caused the problem during one of my earlier debugging sessions. I rebooted the device, set up the tests to run again, and walked out the door.When I got to work the next morning, the first thing I noticed was that the application had crashed again in the same place. By this time, I had access to the source code for the application, and after spending about an hour debugging the problem. The problem turned out to not be in the connection code, but in the graphics code. Every time one of the computer names was selected, the application initiated code that did custom drawing. It was just a small blue flash so that the user would know that the application had recognized the mouse click, much like the way that a button in a Windows-based application appears to sink when pressed. The problem in this application was that every time the test and drawing code ran, there was a resource leak. After a few hundred connections, the resource leak was big enough that the application would crash when the custom drawing code executed.     <\/p>\n<p>I don\u2019t think I ever would have found this bug if I had been writing UI automation that executed functionality without going through the UI directly. I think that in most cases, the best solution for robust UI automation is a method that accesses controls without interacting with the UI, but now, I always keep my eye on any code in the user interface that does more than the functionality that the UI represents. <\/p>\n<\/blockquote>\n<p>To be clear, I am not completely against GUI automation, but I do think that it\u2019s easy (too easy) to get wrong, and that it\u2019s overrated and undervalued among may testers. The best part about my opinion is that if you disagree with me, you have to write <em>good<\/em> GUI automation. In that case, we all win.<\/p>\n<p>&#160;<\/p>\n<p>&#160;<\/p>\n<p>**For the heck of it, I\u2019ve included a screen shot of the app that\u2019s missing from the book (this was the best image I had, and I didn\u2019t think it was good enough to publish).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I\u2019ve stated my distaste for GUI automation before, but after my last post, I feel the need to share some clarification. I\u2019ve been meaning to write this post ever since I wrote GUI Shmooey, so here goes. First off, let me state my main points for disliking GUI automation: It\u2019s (typically) fragile \u2013 tests tend&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-332","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/332","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=332"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/332\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=332"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=332"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=332"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}