{"id":1272,"date":"2018-05-07T09:45:45","date_gmt":"2018-05-07T16:45:45","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=1272"},"modified":"2018-05-07T09:58:07","modified_gmt":"2018-05-07T16:58:07","slug":"the-test-automation-snowman","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/the-test-automation-snowman\/","title":{"rendered":"The Test Automation Snowman"},"content":{"rendered":"<p>I was nothing short of blown away over the past few days, when some comments I made on twitter about UI automation caused a lot of folks to raise their eyebrows.<\/p>\n<p>Here&#8217;s the tweet in question.<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">I&#8217;m not against discussions on the invalidity of the test automation pyramid.<\/p>\n<p>If you don&#8217;t like it, you use whatever model you want as long as it suggests you write AS FEW UI TESTS AS POSSIBLE.<\/p>\n<p>seriously &#8211; stop your infatuation with UI tests<\/p>\n<p>\u2014 Alan Page (@alanpage) <a href=\"https:\/\/twitter.com\/alanpage\/status\/992146080866299904?ref_src=twsrc%5Etfw\">May 3, 2018<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Feedback (blowback?) ranged from accusations of harmful blanket statements to lectures on how what I really meant was &#8220;checking&#8221;, not testing &#8211; with a handful of folks who seemed worried that the world I was describing was too scary compared to the world where they lived. Testing evolves at different speeds in different places, so the last point, at least, was expected. But I stand by the sentiment in that tweet.<\/p>\n<p>The test automation pyramid is a model for thinking about distribution of your tests. Mike Cohn first (I think) <a href=\"https:\/\/www.mountaingoatsoftware.com\/blog\/the-forgotten-layer-of-the-test-automation-pyramid\">wrote about it here.<\/a> In my opinion (and observation), there is an unhealthy obsession in software testing with writing &#8220;automation&#8221; (where &#8220;automation&#8221; means UI automation &#8211; i.e. using tools like Selenium to manipulate the UI to test the system under test). A few folks recently have complained about the pyramid (&#8220;it&#8217;s not a pyramid, it&#8217;s a triangle!&#8221;; &#8220;There are more than 3 types of testing&#8221;, etc.).<\/p>\n<blockquote><p>&#8220;All Models are wrong, some are useful&#8221; &#8212; George Box<\/p><\/blockquote>\n<p>It&#8217;s a good model, with a lot of practical application. Two key takeaways from the pyramid model are:<\/p>\n<ul>\n<li>Write tests at the lowest level where they can find the bug<\/li>\n<li>Minimize the amount of top\/UI level tests<\/li>\n<\/ul>\n<p>I&#8217;m passionate about the second point for three main reasons:<\/p>\n<ol>\n<li>UI Tests are flaky. This was true 25 years ago when I first wrote UI automation, and it&#8217;s true today. They&#8217;re just not as reliable and\u00a0<em>trustworthy<\/em> as lower level tests.<\/li>\n<li>Despite the fact that reliable UI Tests are difficult to write, we (industry) seem to think that UI automation is a reasonable entry point to the world of coding for &#8220;manual&#8221; testers. UI automation is a horrible way to start programming. Shell (or other) scripts to help set up test environments or generate test data would be a much better use of time (and achieve more success) than learning to code by writing UI tests.<\/li>\n<li>UI tests are s l o w. This is fine if you have a handful of tests, but a huge issue if you have hundreds, or thousands of UI tests.<br \/>\n<strong>Note<\/strong> &#8211; if you have a large amount of testing that can only be done at the UI level, that&#8217;s a big red testability issue you should probably address before investing in expensive testing.<\/li>\n<\/ol>\n<p>Let&#8217;s assume for a moment that the problems with UI automation stability have been solved (companies like testim.io have used ML to make some strides in this area, and despite the entry path problem I mentioned above, there is improvement in automation tools and tester skills). If we go with this assumption, then point #1 &#8211; and possibly point #2 above are no longer an issue.<\/p>\n<p>Point #3, however, is not solvable. Tests that automate the UI are slow. Way slow. Like a glacier stuck in molasses slow. I once wrote a UI based networking test to create a folder, share it, connect to it, write files to it, delete files, and then unshare the folder. That test took a little less than two minutes. Problem was, that I needed to test that process for every character possible in isolation (due to issues with DBCS code pages on non-Unicode Windows where details would fill pages of no longer relevant information). On Chinese windows, for example, this was (IIRC), somewhere near 8000 characters.<\/p>\n<p>I wrote an API level test that tested the entire code page &#8211; including varying lengths of folder and share names that ran in under 5 minutes (and less than a minute for Western code pages). Of course, we still did spot checking (both exploratory, and via\u00a0<em>some<\/em> UI automation), but testing at the level closest to where we could find bugs was the most efficient &#8211; both in proximity and speed.<\/p>\n<p>Another view of tests I like is the <a href=\"https:\/\/testing.googleblog.com\/2010\/12\/test-sizes.html\">size model from google<\/a>. Rather than dwell too much on what makes a test a unit or integration test, think of tests in sizes &#8211; where tests of a certain\u00a0<em>duration<\/em> are classified at different levels. This model works well (and solves the pyramid complaints I&#8217;ve seen recently), but it doesn&#8217;t have a visualization.<\/p>\n<p>So &#8211; without further babbling, I created this alternate view &#8211; <strong>The Test Automation Snowman.<\/strong><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-1273 alignnone\" src=\"http:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2018\/05\/snowman.png\" alt=\"\" width=\"563\" height=\"276\" srcset=\"https:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2018\/05\/snowman.png 971w, https:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2018\/05\/snowman-300x147.png 300w, https:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2018\/05\/snowman-768x377.png 768w, https:\/\/angryweasel.com\/blog\/wp-content\/uploads\/2018\/05\/snowman-750x368.png 750w\" sizes=\"auto, (max-width: 563px) 100vw, 563px\" \/><\/p>\n<p>Use it, or ignore it. But I still beg you to consider writing far fewer UI based tests.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I was nothing short of blown away over the past few days, when some comments I made on twitter about UI automation caused a lot of folks to raise their eyebrows. Here&#8217;s the tweet in question. I&#8217;m not against discussions on the invalidity of the test automation pyramid. If you don&#8217;t like it, you use&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-1272","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/1272","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=1272"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/1272\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=1272"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=1272"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=1272"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}