An Approach to Code Coverage

My team at Microsoft is starting to use code coverage a little more diligently. Code coverage has been used for some time on the team, but we’re just now getting to a common approach and recording mechanism. There are a few things about our approach that are a bit different than I’ve seen most other people use, so I thought I’d share it here.

As a quick aside, we measure code coverage using block coverage (block coverage is similar to line coverage except that it groups continuous non-branching statements into a block for counting purposes). I’ve found that bullseye has a very good primer on code coverage measurement for those who are interested in more than I feel like writing about today.

A small group of us worked on the toolset, process and strategy for our team’s use of code coverage. One topic that came up was what target percentage of code coverage we should set as a goal. Years ago, when I was on the CE team, I helped build and deploy our code coverage toolset. Once our tools were working and we measured coverage, we set a goal for the next release (65% if I remember correctly). We cranked up the number a few percentage points each release, and I think we were approaching 80% by the time I left the team. As I type this, I remember something about making a promise to shave my head if we got higher than 80%. That’s only funny because about a year ago, I did shave my head (it’s since grown back).

That history is interesting, because I had been anticipating the question (target for code coverage percentage), and I was (and am) adamant about the code coverage goal I would like our team to shoot for.

I’m adamant that we have no goal for code coverage.

The problem of having a target goal for code coverage is that code coverage will improve – i.e. you get what you measure. Wait a minute – shouldn’t improving code coverage be a good thing? It is (for reasons I’ll explain lower), but here’s what usually happens in practice.

Let’s say I own three features – one major feature with high customer impact, one area with low customer impact, and one that’s somewhere in the middle. Now, let’s assume that my team has a goal of 70% code coverage (measured as an average across the components I own). I measure code coverage, and here are my results.

	Feature 1 – High Impact	Feature 2 – Medium Impact	Feature 3 – Low Impact
Code Coverage	70%	60%	50%

 

Now, if your goal is to get your average to 70% (or even if your goal is a minimum of 70%), you are going to put your testing efforts into testing medium and low impact areas. Shooting for a code coverage goal can convince testers to throw out their prior knowledge of risk and impact, and instead focus on “improving the number” – by testing stuff that probably doesn’t need more testing, while ignoring potentially important stuff. If you never measured code coverage in the first place, would you really spend time doing additional testing on the medium and low impact areas of the product (with more effort needed for the low impact area)?

It’s time to discuss the goal of measuring code coverage. It’s foolish to think that higher code coverage has anything at all to do with product quality. It only measures whether code is executed on at least one path. One of my common “Alan-isms” is this:

The only thing that 80% code coverage tells you is that 20% of your code is completely untested

What code coverage does do is point you to holes in your testing. The goal of measuring code coverage is helping you (as a tester) understand what is not being tested. Once you discover what’s not being tested, you can make a choice based on risk and impact on whether you need to add additional tests – or if your time is better spent elsewhere. By not having a percentage goal for code coverage, I hope the team can focus on improving tests and testing rather than improving a number.

By the way – you can replace the words “code coverage” with “test automation” above and tell a pretty similar story.

Another thing we’re starting to do is measuring code coverage on check-ins. We’re fairly late in the cycle now, so we want to be careful of regressions and ensure that all of the code coming into the product is well tested. What we do is filter our code coverage view to only look at changed lines of code in the changed list. Say we have a binary that has 10k blocks of code, but the latest checkin only changed (or added) 25 blocks. We can filter our testing and coverage on just those 25 blocks and ensure that we’ve at a minimum executed each line at least once during our initial testing. This helps a lot with regressions, as we can ensure that we look at error cases and other little used paths very close to the checkin rather than waiting until those errors pop up at a much later date.

This gives us the potential for 100% code coverage on every changed line of code.

But I would never make that a goal :}