{"id":325,"date":"2011-08-30T16:18:26","date_gmt":"2011-08-30T23:18:26","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=325"},"modified":"2011-08-30T21:40:16","modified_gmt":"2011-08-31T04:40:16","slug":"test-design-for-automation","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/test-design-for-automation\/","title":{"rendered":"Test Design for Automation"},"content":{"rendered":"<p>I\u2019ve been pondering test automation recently. Maybe it\u2019s because of my participation in the upcoming <a href=\"http:\/\/www.softwaretestpro.com\/Event\/1138\">stp summit<\/a> (<em>note<\/em>: shameless self-promotion), but that\u2019s only the surface of it. I\u2019ve complained about misguided test automation efforts before, but it\u2019s more than that too. For every tester that cries out that 100% automation is the only way to test software, someone else is simultaneously stating that only a human (with eyes and a brain) can adequately test software.<\/p>\n<p>The answer, of course, <a href=\"http:\/\/angryweasel.com\/blog\/?p=319\">is in the middle<\/a>.<\/p>\n<p>But I worry that even for those who have figured out that this isn\u2019t an all or nothing proposition, many testers have no idea at all how to <em>design<\/em> an automated test \u2013 which means that they <em>don\u2019t know how to<\/em> <em>design a test in the first place<\/em>. The problem I see most often is in the <em>separation<\/em> of automated and human testing. When approaching a testing problem, you have failed if your first approach is to think about how you\u2019re going to (or not going to) automate. The first step \u2013 and most important \u2013 is to think how you\u2019re going to <em>test. <\/em>From that test design effort, you can deduce what aspects of testing could be accomplished more efficiently with automation (and without).<\/p>\n<p>A common variation of this is the automation of manual tests. It pains me to hear that some testers design scripted test cases, and then automate those actions. This tells me two things: the manual test cases suck, and the automation (probably) sucks. A good human brain-engaged test case never makes a good automated test case, and automating a scripted scenario is rarely a good automated test (although <em>occasionally<\/em>, it\u2019s a start). Some teams even separate the test \u201cwriters\u201d from the \u201cautomators\u201d \u2013 which, to me, is a perfect recipe for crap automation.<\/p>\n<p>An example would be good here. Imagine an application with the following requirements \/ attributes:<\/p>\n<ul>\n<li>The application has a \u201cRoll\u201d button that, when executed, generates five instances of random numbers between 0-9 (inclusive)<\/li>\n<li>The application totals the output from the 5 random numbers and displays them in the \u201cTotal\u201d field.<\/li>\n<li>There are no user editable fields in the application<\/li>\n<\/ul>\n<p>For those of you with no imagination, this is how <em>I<\/em> imagine it.<\/p>\n<p><a href=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2011\/08\/image.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; padding-left: 0px; padding-right: 0px; display: inline; padding-top: 0px; border: 0px;\" title=\"image\" src=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2011\/08\/image_thumb.png\" alt=\"image\" width=\"363\" height=\"194\" border=\"0\" \/><\/a><\/p>\n<p>From a manual only perspective, layout, user experience, and interaction are definitely areas that need to be investigated, and that are usually best done manually. If I were to write a few scripted manual test cases for this (not that I would), they may look something like this:<\/p>\n<p>Test Case 1<\/p>\n<ol>\n<li>Press Roll button<\/li>\n<li>Count (use calculator or an abacas if necessary) to verify that the value in the Total field matches the sum of the values below<\/li>\n<\/ol>\n<p>Test Case 2<\/p>\n<ol>\n<li>Press Roll<\/li>\n<li>Ensure that the values in the lower fields are within 0-9 inclusively<\/li>\n<li>Repeat at least <em>n<\/em> times<\/li>\n<\/ol>\n<p>Test Case 3<\/p>\n<ol>\n<li>Press Roll<\/li>\n<li>Ensure that the value in the top section is between 0 and 45<\/li>\n<li>Repeat<\/li>\n<\/ol>\n<p>I have two complaints about the above tests. The first is that executing them manually is about as exciting as watching a banana rot, and the second is that (as I predicted), they\u2019re not very good automated tests either.<\/p>\n<p>When designing tests, it\u2019s important to think about how automation can (or won\u2019t) make the testing more efficient. What I hope you\u2019ve realized already (and if not, please take a moment to think about the huge testing problem we haven\u2019t talked about yet with this application), is that we\u2019ve done nothing to test for randomness or distribution.<\/p>\n<p>Testing randomness is fun because it\u2019s harder than most people bother to think about. We don\u2019t have to get it perfect for this example, but let\u2019s at least think about it. In addition to the above test cases (granted, the third test case above may be redundant with the first), we need to think about distribution of values within the bottom five boxes. Given the functional goal we\u2019re shooting for, we can probably hit all those test cases and more in a reasonably designed automated test.<\/p>\n<p>Pseudo-code:<\/p>\n<p><code>Loop 100,000 (or more) iterations<br \/>\nPress Roll Button<br \/>\nVerify that sum of output and total field are identical<br \/>\nVerify that output values are between 0 &amp; 9<br \/>\nStore count of values from output boxes 1-5<br \/>\nEnd Loop <\/code><\/p>\n<p>Examine distribution of numbers<br \/>\nExamine sequence of numbers<br \/>\nOther pattern matching activities as needed<\/p>\n<p>The first loop takes care of the main functionality testing, but where automation really helps here is in the analysis of the output. I\u2019d expect a statistically accurate representation of values, repeated values, and sequences.<\/p>\n<p>That\u2019s stuff you want to use automation to solve. Yet, I keep discovering stories of testers who either don\u2019t bother to test stuff like that at all, or put their automation effort into trying to navigate the subtleties of UI testing (yet something else <a href=\"http:\/\/angryweasel.com\/blog\/?p=7\">I have an opinion about<\/a>).<\/p>\n<p>I don\u2019t\u2019 care whether you\u2019re an automator, a tester, a coder, or a <a href=\"http:\/\/angryweasel.com\/blog\/?p=322\">cuttlefish<\/a> \u2013 your goal in testing is not to automate everything, nor is it to validate the user experience through brain-engaged analysis. Your job is to use the most appropriate set of tools and test ideas to carry out your testing mission. You can\u2019t do that when you decide <em>how<\/em> you\u2019re going to test before you start thinking about <em>what<\/em> you\u2019re going to test.<\/p>\n<p><strong>Notes:\u00a0 <\/strong>I don&#8217;t know why I picked 100k for the loop &#8211; it may be overkill, but it&#8217;s also something you have an <em>option<\/em> of doing if you&#8217;re automating the test. I suppose you could do that manually, but you will go insane&#8230;or make a mistake&#8230;or both.<\/p>\n<p>There&#8217;s more to this concept, and follow ups&#8230;maybe. The big point is that I worry that people think that coded tests replace human tests, when they really\u00a0<em>enhance<\/em> human testing &#8211; and also that human testers fail to see where coded tests can help them improve their testing. I think this is a huge problem &#8211; and that it&#8217;s one of those problems that sounds bad, but is actually much, much worse than that&#8230;but I don&#8217;t know how to get that point across.<\/p>\n<p>I should also point out that given my loathing of GUI automation, that I&#8217;d really, really, hope that the pseudo-code I scratched out above could be accomplished by manipulating a model or an object model or the underlying logic directly. I want my logic\/functional tests to test the logic and functionality of the application &#8211; not its flaky UI.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I\u2019ve been pondering test automation recently. Maybe it\u2019s because of my participation in the upcoming stp summit (note: shameless self-promotion), but that\u2019s only the surface of it. I\u2019ve complained about misguided test automation efforts before, but it\u2019s more than that too. For every tester that cries out that 100% automation is the only way to&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-325","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/325","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=325"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/325\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}