{"id":502,"date":"2012-08-23T11:45:00","date_gmt":"2012-08-23T18:45:00","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=502"},"modified":"2012-08-23T11:45:00","modified_gmt":"2012-08-23T18:45:00","slug":"musings-on-test-design","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/musings-on-test-design\/","title":{"rendered":"Musings on Test Design"},"content":{"rendered":"<p>The <a href=\"http:\/\/en.wikipedia.org\/wiki\/Test_automation\">wikipedia entry on test automation<\/a> troubles me. It says:<\/p>\n<blockquote>\n<p>Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.<\/p>\n<\/blockquote>\n<p>I can\u2019t decide whether it bothers me because it\u2019s wrong, or because so many people believe the statement is true. In fact, I know the approach in some organizations is to \u201cdesign\u201d a huge list of manual tests based on written and non-written requirements, and then either try to automate those tests, or pass the tests off to another team (or company) to automate.<\/p>\n<p>This is an awful approach to test design. It\u2019s not holistic. It\u2019s short-sighted. It\u2019s immature, and it\u2019s unprofessional. It\u2019s flat-out crummy testing. I frequently say, \u201cYou should automate 100% of the tests that should be automated\u201d. Let me put it another way to be clear:<\/p>\n<p><strong>Some tests can only be run via some sort of test automation.<\/strong>     <br \/><strong>Some tests can only be done via human interaction.<\/strong><\/p>\n<p>That part (should be) obvious. Here\u2019s the part I don\u2019t think many people get:<\/p>\n<p><strong>You can\u2019t effectively think about automated testing separately from human testing.<\/strong><\/p>\n<p><font color=\"#2e2e2e\">Test Design answers the question, \u201cHow are we going to test this?\u201d The answer to that question will help you decide where automation can help (and where it won\u2019t).<\/font><\/p>\n<p><font color=\"#2e2e2e\">Here\u2019s a screen shot of part of the registration form for outlook.com (disclaimer \u2013 I have no idea how this was tested).<\/font><\/p>\n<p><a href=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2012\/08\/image.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px\" title=\"image\" border=\"0\" alt=\"image\" src=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2012\/08\/image_thumb.png\" width=\"429\" height=\"401\" \/><\/a><\/p>\n<p>Let\u2019s look at two different ways of answering the \u201cHow will we test this?\u201d question.<\/p>\n<p>The \u201cautomator\u201d may look at this and think the following.<\/p>\n<ul>\n<li>I\u2019ll build a table of first and last names and use those for test accounts<\/li>\n<li>I\u2019ll try every combination of Days, Months, and Years for Birthdate<\/li>\n<li>I\u2019ll generate a bunch of different test account names<\/li>\n<li>I\u2019ll create a password for each account <\/li>\n<li>Once I try all of the combinations, I can sign off on this form<\/li>\n<li>(or, they may think, \u201cI wonder what sort of test script the test team will ask me to automate\u201d<\/li>\n<\/ul>\n<p>The \u201chuman\u201d may look at the same form and think this:<\/p>\n<ul>\n<li>I\u2019ll want to try short names, long names, and blank names<\/li>\n<li>I\u2019ll see if I can find invalid dates (e.g. Feb 29 in a non-leap year)<\/li>\n<li>Some characters are invalid for email names \u2013 I\u2019ll try to find some of those<\/li>\n<li>I\u2019ll make sure the 8-character minimum and case sensitivity is enforced<\/li>\n<li>Oh \u2013 I\u2019ll try that passwords with foreign characters too.<\/li>\n<li>Once I go through all of that, and anything else I discover, I can sign off on this form.<\/li>\n<\/ul>\n<p>I\u2019ll fire off a disclaimer now, because I\u2019ve probably pissed off both \u201cautomators\u201d, and \u201chumans\u201d with the generalizations above. I know there\u2019s overlap. My argument is that there should be more.<\/p>\n<p>In my contrived examples above, the \u201cautomator\u201d is answering the question, \u201cWhat can I automate?\u201d, and the \u201chuman is answering the question, \u201cWhat can I explore or discover?\u201d. Neither is directly answering the question, \u201chow are we going to test this?\u201d.<\/p>\n<p>I could just merge the lists, but that\u2019s not exactly right. Let\u2019s throw away humans and coders for a minute and see if we can use a mind map to get the whole picture together. Here\u2019s what I came up with giving myself a limit of 10 minutes. There\u2019s likely something missing, but it\u2019s a starting point.<\/p>\n<p><a href=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2012\/08\/RegistrationForm.png\"><img loading=\"lazy\" decoding=\"async\" style=\"background-image: none; border-bottom: 0px; border-left: 0px; padding-left: 0px; padding-right: 0px; display: inline; border-top: 0px; border-right: 0px; padding-top: 0px\" title=\"Registration Form\" border=\"0\" alt=\"Registration Form\" src=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2012\/08\/RegistrationForm_thumb.png\" width=\"576\" height=\"300\" \/><\/a><\/p>\n<p>Now (and at no time before now), I can begin to think about where automation may help me. My goal isn\u2019t to automate everything, it\u2019s to automate what makes the most sense to automate. Things like submitting the form from multiple machines at once, or cycling through tables of data make sense to automate early. Other items, like checking for non-existent days or checking max length of a name are nice exploratory tasks. And then, there are ideas like foreign characters in the name, or trying RTL languages that may <em>start<\/em> as an exploratory test, but may lead to ideas for automation.<\/p>\n<p>The point is, and this is worth repeating so often, is that thinking of test automation and \u201chuman\u201d testing as separate activities is one of the biggest sins I see in software testing today. It\u2019s not only ineffective, but I think this sort of thinking is a detriment to the advancement of software testing. <\/p>\n<p>In my world, there are no such things as automated testing, exploratory testing, manual testing, etc.<\/p>\n<p>There is only testing. <\/p>\n<p>More musings and rants to follow.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The wikipedia entry on test automation troubles me. It says: Commonly, test automation involves automating a manual process already in place that uses a formalized testing process. I can\u2019t decide whether it bothers me because it\u2019s wrong, or because so many people believe the statement is true. In fact, I know the approach in some&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-502","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/502","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=502"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/502\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}