Code Coverage with Manual Testing
Here’s a tip. Want me to write about something in particular? Ask me a question. It’s that easy. Tim Coulter had a question about my last blog post. He asked:
Would love to hear how you check code coverage of manual testing, actually. Is that just “feature coverage” (break it up into logical blocks and test those)?
I realized that a lot of people may not be measuring code coverage during manual testing – but it’s quite valuable to do so, as measuring code coverage still helps you find missing tests.
Let’s consider a silly command line app that does math operations. It takes 3 parameters, the first two are the values to operate on, and the third parameter is the operator to execute. For example, a command line of 2 3 + would return a value of 5. Let’s test it (manually).
c:\example>calul8r.exe Usage: calcul8r [value] [value] [operation] for example: calcul8r 4 2 + c:\example>calul8r.exe 2 3 + The result of 2 + 3 is 5 c:\example>calul8r.exe 4 2 - The result of 4 - 2 is 2 c:\example>calul8r.exe 5 5 * The result of 5 * 5 is 25 c:\example>calul8r.exe 15 3 / The result of 15 / 3 is 5
OK – looks like it sort of works. But are there test cases we forgot? Let’s look at what code coverage tells us.
1 int DoMath(int val1, int val2, int valOperator)
2 {
3 int result = 0xbaadf00d;
4 switch (valOperator)
5 {
6 case ‘+’:
7 result = val1 + val2;
8 break;
9 case ‘x’:
10 case ‘*’:
11 result = val1 * val2;
12 break;
13
14 case ‘/’:
15 case ‘\\’:
16 if (val2 ==0)
17 {
18 printf("Invalid input. Cannot divide by zreo");
19 }
20 else
21 {
22 result = val1 / val2;
23 }
24 break;
25
26 case ‘-‘:
27 result = val1 – val2;
28 break;
29
30 default:
31 printf("Unknown oprator – %d\n");
32 break;
33 }
34 return result;
35 }
The “meat” of this app is in the DoMath function, and the results show that I missed a few lines of code in my first round of tests. The first two lines I missed (lines 9 and 15) are a bit of discovery. It looks like the app can take the letter x as well as an * (asterisk) for multiplication, and a backslash (\) as well as a forward slash (/) for division. The functionality is the same, but it’s interesting nonetheless.
Oh – but I was stupid and forgot to test for divide by zero. It’s nice that there is some error handling for that, but if you look closer, you’ll see that executing this particular test would reveal a typo in the output string.
I also forgot to test with an invalid operator – and wouldn’t you know it, there’s another typo there. What I’ve done is used the code coverage data to guide my design for new tests. This app is simple, but the idea works on a larger scale too. I’ve known testers to discover large areas of functionality they weren’t aware of before, or (more often) hidden functionality within the areas they had tested extensively.
An important thing to remember is that once I’ve added those test cases (and reported those bugs), is that I’ll have 100% code coverage. Remember though, that 100% code coverage doesn’t mean bug free. In fact, this app doesn’t check for integer overflow. Sure enough, check this out:
c:\example>calul8r.exe 2147483647 1 + The result of 2147483647 + 1 is -2147483648
Oops! Remember – just because the code is “covered”, it doesn’t mean it’s tested. Please don’t forget that (and don’t let your managers forget either).
Thanks Tim, for prompting this post – I hope you (and others) find it helpful.
This is nice little piece for showing how code coverage can be measured for a manual test of a simple command line based calculator. But how do I answer the question for “what’s the coverage of your testing?” for a multiple component-based application, consisting of a C/C++ major component, an application server, and customized business logic? Is there any way to get down to the code coverage level from a business perspective, or is feature/business rule coverage more appropriate here?
I am looking for a tool that measures code coverage of manual testing for a PHP application. Similar to how XDebug can measure the code coverage of unit tests and produces an output, I would like to be able to tell my environment when to begin measuring code coverage, perform manual testing on a development website(i.e. login and place an order as a registered customer), end measuring code coverage, and produce an output similar to what you have shown in this article.
Are there any tools that you can recommend that do this for PHP websites? I have only seen a tool like this that is part of Visual Studio.
Sorry – no idea. Bing may be of some help.