{"id":342,"date":"2011-09-13T16:44:31","date_gmt":"2011-09-13T23:44:31","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=342"},"modified":"2011-09-13T16:44:31","modified_gmt":"2011-09-13T23:44:31","slug":"its-probably-a-design-problem","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/its-probably-a-design-problem\/","title":{"rendered":"It&rsquo;s (probably) a Design Problem"},"content":{"rendered":"<p>As you know, I occasionally rant about GUI automation; but I don\u2019t think I\u2019ve done a good job explaining some of the reasons why it <em>can<\/em> be fragile, and when it can actually be a good idea.<\/p>\n<p>Let\u2019s take a deeper look at some of the attributes of the GUI automation design challenges.<\/p>\n<ul>\n<li><strong>Record &amp; Playback automation.<\/strong> This is a non-starter for me. I\u2019ve never seen recorded automation work well in the long term\u2013 if your automation is based on record &amp; playback, I can\u2019t imagine it being successful. Yes, the tools are getting better, but the real problem is \u2026<\/li>\n<li><strong>What are you going to do about the oracle? \u2013 <\/strong>Regardless of whether you write the automation, or you record it, simple playback of a user workflow has an oracle problem. The failure model in many UI tests is that nothing \u201cbad\u201d happens \u2013 the problem is that you don\u2019t often know what flavors of \u201cbad\u201d you\u2019re looking for, so your tests may miss stuff<strong>. <\/strong><em>However, <\/em>if you <em>design<\/em> your tests with a trusted oracle (meaning if the test fails, you <u>know<\/u> there\u2019s a problem, and if it passes, you <u>know<\/u> the scenario works), you probably have usable UI automation. By the way \u2013 if your oracle solution involves screen comparisons, I think you\u2019re heading down a scary path \u2013 please consider this solution as a last, end-of-the-world type solution.<\/li>\n<li><strong>Bad Test Design I \u2013 <\/strong>Many authors of UI tests seemingly fail to realize they\u2019re writing automation. Basic verification that would be hit by anyone walking through the basics of the application isn\u2019t worth much to me. However, if your automation actually <em>takes advantage of automation<\/em> \u2013 meaning that loops, randomness, input variations, and loads of other ideas are part of the automation approach, the approach may be worthwhile.<\/li>\n<li><strong>Bad Test Design II <\/strong>\u2013 I dislike tests that do exactly the same thing every time. While valuable for regression testing, \u201chard coded\u201d tests have severe limitations. In notepad, I can open a file by clicking the file menu, then selecting open, or I can press Alt-f, then o on the keyboard, I can press Ctrl-o, I can drag a file onto an open instance of notepad, or I can double click a file associated with notepad. A <em>well-designed<\/em> notepad test could just state File.Open(<em>filename<\/em>), and then use a randomly selected file open method. Too much UI automation doesn\u2019t have this sort of abstraction and limits the effectiveness of the approach.<\/li>\n<li><strong>Bad Test Design III<\/strong> \u2013 Lack of forward thinking is another design flaw I see often. I once saw a GUI test suite that ran (successfully) on over 20 different localized versions of Windows 95, including RTL languages \u2013 <em>and<\/em> it ran successfully on Windows 98. Unfortunately, not all test authors consider the variety of scenarios where their tests <em>could<\/em> run.<\/li>\n<li><strong>Bad Test Design IV<\/strong> \u2013 Failure to consider what can fail. This one irks me, because testers <em>should know that bad things can happen<\/em>. Consider what will happen when someone changes window text, a control type, or dialog layout. Consider what happens when Windows reboots your machine in the middle of a test run. You don\u2019t necessarily have to make your test resilient to these changes, but at the very least, you need to make the error text point to exactly what changed and caused the error. Too often, tests fail and give zero information on why they failed. Plan for failure and ensure that all test failures tell you <u>exactly<\/u> what is wrong.<\/li>\n<li><strong>Bad Test Design V<\/strong> \u2013 As a UI tester, you should have some notion of what sorts of UI automation are reliable, and which sorts are flaky. Using SendKeys works\u2026but it probably should be a last resort. Good UI automation means that you know at least three ways to accomplish any task, and know which of the approaches is most reliable and which is least reliable.<\/li>\n<li><strong>Bad Test Design VI<\/strong> \u2013 One of my test design smells is Sleep (or similar) statements. The more I see in your test code, the less I trust it. Repeat after me \u2013 \u201cSleep functions are not a form of synchronization\u201d. Most frameworks have an alternative to Sleep. There is always a better alternative to Sleep statements.<\/li>\n<li><strong>Fragile UI \u2013<\/strong> It\u2019s really easy to write automation that fails anytime the UI changes. It\u2019s also really easy to write UI that breaks automation. If you don\u2019t start testing until late in the product cycle, and know that the UI automation won\u2019t be used in subsequent versions of the product, an investment in UI automation may make sense (given that it solves a real testing problem. Alternatively, if you\u2019re involved early in the product cycle, you could <em>wait<\/em> to write UI automation until the UI was complete. A third (and recommended) approach is to make sure that the application under test <em>is designed<\/em> in a way that makes UI automation more robust (i.e. increase testability). Testability is, in short, the expense of test. Rewriting automation frequently during the product cycle is expensive. Writing complex oracles to verify things that <em>should<\/em> be easier is expensive. Working with developers to implement a UI model that enables straightforward and sustainable automation is cheap&#160; &#8211; especially when you consider the long term benefits and gains.I probably have another post on this subject (testability) alone.<\/li>\n<\/ul>\n<p>My annoyance with GUI automation isn\u2019t a blanket view. The problem is that few teams put the effort into product and test design that successful GUI automation requires. I think GUI automation <em>can<\/em> be awesome \u2013 but we (as an industry) don\u2019t seem to care enough\u2026yet.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As you know, I occasionally rant about GUI automation; but I don\u2019t think I\u2019ve done a good job explaining some of the reasons why it can be fragile, and when it can actually be a good idea. Let\u2019s take a deeper look at some of the attributes of the GUI automation design challenges. Record &amp;&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-342","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/342","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=342"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/342\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=342"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=342"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=342"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}