{"id":868,"date":"2014-05-07T12:40:31","date_gmt":"2014-05-07T19:40:31","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=868"},"modified":"2014-05-07T12:40:33","modified_gmt":"2014-05-07T19:40:33","slug":"stop-writing-automation","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/stop-writing-automation\/","title":{"rendered":"Stop Writing Automation"},"content":{"rendered":"

After releasing The A Word<\/a>, I didn\u2019t plan on writing any more posts about automation. But, after pondering transitions in test, and after reading this post<\/a> from Noah Sussman, I have a thought in my head that I need to share.<\/p>\n

I don\u2019t think testers should write automation. \"Star\"<\/strong><\/p>\n

I suppose I better explain myself. <\/p>\n

All automation isn\u2019t created equal. Automation works wonderfully for short confirmatory or validation tests. Unit, functional, acceptance, integration tests, and all other \u201cshort\u201d tests lend themselves very well to automation. But I think it\u2019s wasteful and inefficient to have testers write this automation \u2013 this should be written by the code owners. Testing your own code (IMO) improves design, prevents regression, and takes much, much less time than passing code off to another team to test.<\/p>\n

That leaves the test team to write automation for end-to-end scenarios. There\u2019s nothing wrong with that\u2026except that writing end to end automated tests is hard (especially, as Noah points out<\/a>, at the GUI level). The goal of automation (as touted by many of the vendors), is to enable running a bunch of tests automatically, so that testers will have more time for hands-on testing. In reality, I think that most test teams spend half of their time writing automation, half of their time maintaining and debugging automation, and half of their time doing everything else.<\/p>\n

Let\u2019s look at a typical scenario.<\/p>\n

\"image\"<\/a><\/p>\n

Pretty easy \u2013 you only need to automate three actions, add a bit of validation and you\u2019re done!<\/p>\n

If you\u2019ve attempted something like this before, you know that\u2019s not the whole story. A good automated test doesn\u2019t just execute a set of user actions and then go on it\u2019s way \u2013 you need to look at error conditions, and try to write the test in a way that prevents it from breaking.<\/p>\n

\"image\"<\/a><\/p>\n

It\u2019s hard. It\u2019s fragile. It\u2019s a pain in the ass. So I say, stop doing it. <\/p>\n

\u201cBut Alan \u2013 we have<\/em> to test the scenario so that we know it works for our customers\u201d. I believe that you want to know that the scenario works \u2013 but as much as you try to be the customer, nobody is a better representative of the customer than the customer.<\/em> And \u2013 validating scenarios is quite a bit easier if we let the customers do it. <\/p>\n

NOTE: I acknowledge that for some contexts, letting the customers generate usage data for an untested scenario is an inappropriate choice. But even when I\u2019ve already tested the scenario, I still want (need!) to know what the customer experience is. As a tester, I can be the voice of the customer, but I am NOT <\/em>the customer.<\/p>\n

Now \u2013 let\u2019s look at the same scenario:<\/p>\n

\"image\"<\/a><\/p>\n

But this time, instead of writing an automated test, I\u2019ve added code (sometimes called instrumentation) to the product<\/em> code that let\u2019s me know when I\u2019ve taken an action related to the scenario. While we\u2019re at it, let\u2019s add more instrumentation at every error path in the code.<\/p>\n

\"image\"<\/a><\/p>\n

Now, when a user attempts the scenario, you could get logs something like this:<\/p>\n

05072014-1601-3143: Search started for FizzBuzz\n05072014-1601-3655: Download started for FizzBuzz\n05072014-1602-2765: Install started for FizzBuzz\n05072014-1603-1103: FizzBuzz launched successfully\n<\/pre>\n
or <\/p>\n
05072014-1723-2143: Search started for FizzBuzz\n05072014-1723-3655: Download started for FizzBuzz\n05072014-1723-2945: ERROR 115: Connection Lost\n<\/pre>\n
or <\/p>\n
05072014-1819-3563: Search started for FizzBuzz\n05072014-1819-3635: ERROR 119: Store not available.\n<\/pre>\n
 <\/p>\n
From this, you can begin to evaluate scenario success by looking at how many people get through the entire scenario, and how many fail for particular errors.<\/p>\n
\"image\"<\/a><\/p>\n
or generate data like this:<\/p>\n
\"image\"<\/a><\/p>\n
And now we have information about our product that\u2019s directly <\/em>related to real customer usage, and the code to enable it is substantially more simple and easy to maintain than the traditional scenario automation.<\/p>\n
\"Star\"A more precise way to put, I don\u2019t think testers should write automation.<\/strong> is \u2013 I think that many testers in some contexts don\u2019t need to write automation anymore. Developers should write more<\/em> automation, and testers should try to learn more from the product in use. Use this approach in financial applications at your own risk!<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"
After releasing The A Word, I didn\u2019t plan on writing any more posts about automation. But, after pondering transitions in test, and after reading this post from Noah Sussman, I have a thought in my head that I need to share. I don\u2019t think testers should write automation. I suppose I better explain myself. All…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-868","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/868","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=868"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/868\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=868"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=868"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=868"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}