I’m going to do one more (I think) post on this subject. Markus asked the following about my latest post:
But how do I answer the question for “what’s the coverage of your testing?” for a multiple component-based application, consisting of a C/C++ major component, an application server, and customized business logic?
The answer is sort of in this post, but it’s worth elaboration.
First and foremost, if someone asks “what’s the coverage of your testing”, you can answer in multiple ways. You can say “we’ve tested the requirements”, or “we have an average of 70% line coverage throughout the product”, or “we’ve covered all of the key customer scenarios”. But what you probablyshould ask is, “What do you really want to know?” Explain that “testing coverage” probably isn’t the best way to think about it, and “identifying missing tests” is a better train of thought.
The sample I used was a simple command line app, but we usually test much larger systems, so let me also explain how you could scale code coverage from my previous example to a larger application. The solution I prefer isn’t complex – you just start at a high level, then drill down until you get to something actionable. Let’s say my overall product code coverage is 60%. That tells me that 40% of my code is untested. For the sake of easy math, let’s say the application has a million lines of code. That’s 400,000 untested lines of code. Which ones do we look at first?
Let’s drill down into the four main feature areas of the (fictitious) app.
| Feature Area | Priority | Lines of Code | Code Coverage |
| UI Controls | 3 | 100,000 | 60% |
| Engine | 1 | 400,000 | 60% |
| Web Server | 2 | 400,000 | 70% |
| Utilities | 4 | 100,000 | 50% |
OK – I notice that the Utilities only have 50% code coverage, but it’s the lowest priority, so I won’t start there. Instead, I’m going to look for testing holes in our pri 1 area (Engine). Let’s dig in again.
| Engine Drill Down | Lines of Code | Code Coverage |
| Core | 200000 | 60% |
| File Access | 75000 | 75% |
| Protocols | 50000 | 53% |
| Filters | 75000 | 56% |
Protocols are the worst (although we’ll definitely need to look at Filters as well at some point). Digging in further tells us …
| Protocols Drill Down | Lines of Code | Code Coverage |
| XML Transform | 20000 | 80% |
| Format Codes | 10000 | 55% |
| Models | 20000 | 40% |
Looking like I’m missing test cases for Models…
| Models Drill Down | Lines of Code | Code Coverage |
| File1.cpp | 1000 | 5% |
| File2.cpp | 1000 | 70% |
| File3.cpp | 2000 | 70% |
| File4.cpp | 500 | 60% |
| File5.cpp | 500 | 60% |
oh wow – I’ve barely tested the functionality in one of the files (File1.cpp). One step deeper confirms the story.
| File1.cpp Drill Down | Code Coverage |
| SomeFunc | 10% |
| SomeOtherFunc | 0% |
| SomePeculiarFunc | 0% |
| ImportantFunc | 3% |
From here, I’d look at each of the functions in this file and see what testing I’m missing that may help me hit this code. Once again, all I’ve done is used the code coverage data to guide my testing. Once I’ve looked at what I may be missing in File1.cpp, I may back out and look at Format Codes, or I may back all the way back to looking at Filters. I still use my knowledge of risk and priority to tell me where to investigate, then use the coverage data as a tool to help guide me in the right direction.
An important thing to reiterate, is that I’m still not concerned with improving the code coverage number – I just want to use the data to see what I’m missing. Sure, the number will go up anyway, but it’s not the point. The point is reducing risk by identifying missing test areas.