{"id":172,"date":"2010-08-09T13:57:05","date_gmt":"2010-08-09T20:57:05","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=172"},"modified":"2010-08-09T14:19:44","modified_gmt":"2010-08-09T21:19:44","slug":"an-approach-to-code-coverage","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/an-approach-to-code-coverage\/","title":{"rendered":"An Approach to Code Coverage"},"content":{"rendered":"

My team at Microsoft is starting to use code coverage a little more diligently. Code coverage has been used for some time on the team, but we\u2019re just now getting to a common approach and recording mechanism. There are a few things about our approach that are a bit different than I\u2019ve seen most other people use, so I thought I\u2019d share it here.<\/p>\n

As a quick aside, we measure code coverage using block coverage (block coverage is similar to line coverage except that it groups continuous non-branching statements into a block for counting purposes). I\u2019ve found that bullseye<\/a> has a very good primer on code coverage measurement for those who are interested in more than I feel like writing about today.<\/p>\n

A small group of us worked on the toolset, process and strategy for our team\u2019s use of code coverage. One topic that came up was what target percentage of code coverage we should set as a goal. Years ago, when I was on the CE team, I helped build and deploy our code coverage toolset. Once our tools were working and we measured coverage, we set a goal for the next release (65% if I remember correctly). We cranked up the number a few percentage points each release, and I think we were approaching 80% by the time I left the team. As I type this, I remember something about making a promise to shave my head if we got higher than 80%. That\u2019s only funny because about a year ago, I did shave my head (it\u2019s since grown back).<\/p>\n

That history is interesting, because I had been anticipating the question (target for code coverage percentage), and I was (and am) adamant about the code coverage goal I would like our team to shoot for.<\/p>\n

I\u2019m adamant that we have no goal for code coverage<\/span>.<\/strong><\/p>\n

The problem of having a target goal for code coverage is that code coverage will improve \u2013 i.e. you get what you measure. Wait a minute \u2013 shouldn\u2019t improving code coverage be a good thing? It is (for reasons I\u2019ll explain lower), but here\u2019s what usually happens in practice.<\/p>\n

Let\u2019s say I own three features \u2013 one major feature with high customer impact, one area with low customer impact, and one that\u2019s somewhere in the middle. Now, let\u2019s assume that my team has a goal of 70% code coverage (measured as an average across the components I own). I measure code coverage, and here are my results.<\/p>\n\n\n\n\n
\u00a0<\/td>\nFeature 1 \u2013 High Impact<\/td>\nFeature 2 \u2013 Medium Impact<\/td>\nFeature 3 \u2013 Low Impact<\/td>\n<\/tr>\n
Code Coverage<\/td>\n70%<\/td>\n60%<\/td>\n50%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

\u00a0<\/p>\n

Now, if your goal is to get your average to 70% (or even if your goal is a minimum of 70%), you are going to put your testing efforts into testing medium and low impact areas<\/em>. Shooting for a code coverage goal can convince testers to throw out their prior knowledge of risk and impact, and instead focus on “improving the number” – by testing stuff that probably doesn’t need more testing, while ignoring potentially important stuff. If you never measured code coverage in the first place, would you really spend time doing additional testing on the medium and low impact areas of the product (with more effort needed for the low impact area)?<\/p>\n

It\u2019s time to discuss the goal of measuring code coverage. It\u2019s foolish to think that higher code coverage has anything at all to do with product quality. It only measures whether code is executed on at least one path. One of my common \u201cAlan-isms\u201d is this:<\/p>\n

The only thing that 80% code coverage tells you is that 20% of your code is completely<\/span> untested<\/p><\/blockquote>\n

What code coverage does<\/em> do is point you to holes in your testing. The goal of measuring code coverage is helping you (as a tester) understand what is not being tested. Once you discover what\u2019s not being tested, you can make a choice based on risk and impact on whether you need to add additional tests \u2013 or if your time is better spent elsewhere. By not having a percentage goal for code coverage, I hope the team can focus on improving tests and testing rather than improving a number.<\/p>\n

By the way \u2013 you can replace the words \u201ccode coverage\u201d with \u201ctest automation\u201d above and tell a pretty similar story.<\/p>\n

Another thing we\u2019re starting to do is measuring code coverage on check-ins. We\u2019re fairly late in the cycle now, so we want to be careful of regressions and ensure that all of the code coming into the product is well tested. What we do is filter our code coverage view to only look at changed lines of code in the changed list. Say we have a binary that has 10k blocks of code, but the latest checkin only changed (or added) 25 blocks. We can filter our testing and coverage on just those 25 blocks and ensure that we\u2019ve at a minimum executed each line at least once during our initial testing. This helps a lot with regressions, as we can ensure that we look at error cases and other little used paths very close to the checkin rather than waiting until those errors pop up at a much later date.<\/p>\n

This gives us the potential<\/em> for 100% code coverage on every changed line of code.<\/p>\n

But I would never make that a goal :}<\/p>\n","protected":false},"excerpt":{"rendered":"

My team at Microsoft is starting to use code coverage a little more diligently. Code coverage has been used for some time on the team, but we\u2019re just now getting to a common approach and recording mechanism. There are a few things about our approach that are a bit different than I\u2019ve seen most other…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-172","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/172","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=172"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/172\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=172"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=172"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=172"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}