{"id":362,"date":"2011-11-08T05:34:41","date_gmt":"2011-11-08T13:34:41","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=362"},"modified":"2011-11-08T21:35:32","modified_gmt":"2011-11-09T05:35:32","slug":"test-innovation","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/test-innovation\/","title":{"rendered":"Test Innovation"},"content":{"rendered":"

I\u2019m in Frankfurt for a few days, and tomorrow (Wednesday), I am giving a keynote at German Testing Day<\/a> on Test Innovation at Microsoft<\/em>.<\/p>\n

At this point, I\u2019ll give you a few moments to chuckle, laugh, or make snide remarks about the topic. I think I\u2019ve heard it all before; \u201ctesters at MS just write automation all day\u201d, \u201ctesters at Microsoft don\u2019t care about quality\u201d, or \u201ctesters at MS don\u2019t get testing\u201d (all quotes I\u2019ve heard or read about MS testing \u2013 and all, interestingly, by people who have never worked at Microsoft). This post is, I suppose, inspired by my talk tomorrow, but is more of a general example than a reflection of the topics of tomorrow\u2019s presentation.<\/p>\n

I think our lack of open-ness in how we test and innovate in testing at Microsoft is (IMO) one of our biggest flaws as a company \u2013 or at least one of our biggest flaws in being a good member of the worldwide testing community. We tried to make a dent with testercenter<\/a>, but that project is all but dead. A few of us who test at MS share what we do, but none of us do a great job sharing our innovations. I hope we can get better at that soon. The short and honest truth is that I can easily name at least fifty testers at MS whose testing skills and ability to innovate are equal or better than any tester you know. I suppose in a company of nearly 10k testers that\u2019s not really a big deal, but it\u2019s really hit home as I\u2019ve prepared for this presentation. I have some ideas for getting some of these people and their ideas on the global radar, but for now, you\u2019ll just have to trust me.<\/p>\n

I\u2019m probably not the right person to lecture (or write) about innovation, but I can tell you what kinds of innovative testing solutions tend to be successful (at MS at least). A lot of innovation starts with the \u201cThere\u2019s got to be a better way\u201d approach. The worst rut to get into in software development is the \u201cwe\u2019ve always done it that way\u201d syndrome. Innovative testers stop, look at the big picture, and ask, ‘\u201dis there a better way to do this?\u201d Some of the biggest test innovations at MS have been solutions that take something complex or difficult (or \u201cimpossible\u201d), and making it brain dead simple. One tester created automatic filtering of code coverage reports based on a specific change set \u2013 in other words, he made it brain dead simple to know if you (or any tester) tested each changed line in a specific fix or feature addition. Some of our games testers have created a system that allow the bug triage team to navigate to the exact place in the game where an error occurs from the bug tracking system <\/em>(and conversely, the ability to create a bug from the place in the game where the tester observed the error).<\/p>\n

Other innovation happens when someone finds a way to implement a known idea into a testing process. Testing localization is a huge (and hugely expensive) task. Ross Smith (and a few of his cohorts) had the idea<\/a> to optimize the task via crowdsourcing and gaming elements.The results have been phenomenal.<\/p>\n

Eric Sevareid once said,<\/p>\n

The chief cause of problems is solutions.<\/p><\/blockquote>\n

Many years ago, test systems at MS reached a level of maturity and scale that enabled teams to easily run millions<\/em> of tests. These systems[1] were the culmination of several innovations, but eventually led to other problems. The prime example of a problem created by the test system innovation is the challenge of what to do with all of those failed tests. Think for a moment. If you run a million tests and have a 99% pass rate (I know, but this is just an example), you have ten-thousand<\/em> failures to investigate. Automatic failure analysis (discussed in HWTSAM<\/a>) is an innovation that greatly reduces the time testers need to investigate automation failures (my stance is that automation is worthless unless every aspect of automation beyond test authoring is not also automated). We\u2019ve also made great strides in test selection (let\u2019s say you only had time to run five thousand of those million tests \u2013 which would you run?).<\/p>\n

Of course, innovation occurs in the testing itself (as well as in the systems surrounding the testing). We\u2019ve done amazing things with test generation, test design, and run time analysis.<\/p>\n

Some of which I\u2019ll share with the audience tomorrow \u2013 and likely in this blog sometime in the future.<\/p>\n

 <\/p>\n

[1] I described a test automation system in Beautiful Testing<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

I\u2019m in Frankfurt for a few days, and tomorrow (Wednesday), I am giving a keynote at German Testing Day on Test Innovation at Microsoft. At this point, I\u2019ll give you a few moments to chuckle, laugh, or make snide remarks about the topic. I think I\u2019ve heard it all before; \u201ctesters at MS just write…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-362","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/362","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=362"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/362\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=362"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=362"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=362"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}