{"id":176,"date":"2010-08-12T03:46:24","date_gmt":"2010-08-12T10:46:24","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=176"},"modified":"2010-08-12T03:46:24","modified_gmt":"2010-08-12T10:46:24","slug":"scaling-code-coverage","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/scaling-code-coverage\/","title":{"rendered":"Scaling Code Coverage"},"content":{"rendered":"

I\u2019m going to do one more (I think) post on this subject. Markus asked the following about my latest post<\/a>:<\/p>\n

\n

But how do I answer the question for \u201cwhat\u2019s the coverage of your testing?\u201d for a multiple component-based application, consisting of a C\/C++ major component, an application server, and customized business logic?<\/p>\n<\/blockquote>\n

The answer is sort of<\/em> in this post<\/a>, but it\u2019s worth elaboration.<\/p>\n

First and foremost, if someone asks \u201cwhat\u2019s the coverage of your testing\u201d, you can answer in multiple ways. You can say \u201cwe\u2019ve tested the requirements\u201d, or \u201cwe have an average of 70% line coverage throughout the product\u201d, or \u201cwe\u2019ve covered all of the key customer scenarios\u201d. But what you probablyshould <\/em>ask is, \u201cWhat do you really want to know?<\/strong>\u201d Explain that \u201ctesting coverage\u201d probably isn\u2019t the best way to think about it, and \u201cidentifying missing tests\u201d is a better train of thought.<\/p>\n

The sample I used was a simple command line app, but we usually test much larger systems, so let me also explain how you could scale code coverage from my previous example to a larger application. The solution I prefer isn\u2019t complex \u2013 you just start at a high level, then drill down until you get to something actionable. Let\u2019s say my overall product code coverage is 60%. That tells me that 40% of my code is untested. For the sake of easy math, let\u2019s say the application has a million lines of code. That\u2019s 400,000 untested lines of code. Which ones do we look at first?<\/p>\n

Let\u2019s drill down into the four main feature areas of the (fictitious) app.<\/p>\n\n\n\n\n\n\n\n
Feature Area<\/td>\nPriority<\/td>\nLines of Code<\/td>\nCode Coverage<\/td>\n<\/tr>\n
UI Controls<\/td>\n3<\/td>\n100,000<\/td>\n60%<\/td>\n<\/tr>\n
Engine<\/td>\n1<\/td>\n400,000<\/td>\n60%<\/td>\n<\/tr>\n
Web Server<\/td>\n2<\/td>\n400,000<\/td>\n70%<\/td>\n<\/tr>\n
Utilities<\/td>\n4<\/td>\n100,000<\/td>\n50%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

OK \u2013 I notice that the Utilities only have 50% code coverage, but it\u2019s the lowest priority, so I won\u2019t start there. Instead, I\u2019m going to look for testing holes in our pri 1 area (Engine). Let\u2019s dig in again.<\/p>\n\n\n\n\n\n\n\n
Engine Drill Down<\/td>\nLines of Code<\/td>\nCode Coverage<\/td>\n<\/tr>\n
Core<\/td>\n200000<\/td>\n60%<\/td>\n<\/tr>\n
File Access<\/td>\n75000<\/td>\n75%<\/td>\n<\/tr>\n
Protocols<\/td>\n50000<\/td>\n53%<\/td>\n<\/tr>\n
Filters<\/td>\n75000<\/td>\n56%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Protocols are the worst (although we\u2019ll definitely need to look at Filters as well at some point). Digging in further tells us \u2026<\/p>\n\n\n\n\n\n\n
Protocols Drill Down<\/td>\nLines of Code<\/td>\nCode Coverage<\/td>\n<\/tr>\n
XML Transform<\/td>\n20000<\/td>\n80%<\/td>\n<\/tr>\n
Format Codes<\/td>\n10000<\/td>\n55%<\/td>\n<\/tr>\n
Models<\/td>\n20000<\/td>\n40%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Looking like I\u2019m missing test cases for Models\u2026<\/p>\n\n\n\n\n\n\n\n\n
Models Drill Down<\/td>\nLines of Code<\/td>\nCode Coverage<\/td>\n<\/tr>\n
File1.cpp<\/td>\n1000<\/td>\n5%<\/td>\n<\/tr>\n
File2.cpp<\/td>\n1000<\/td>\n70%<\/td>\n<\/tr>\n
File3.cpp<\/td>\n2000<\/td>\n70%<\/td>\n<\/tr>\n
File4.cpp<\/td>\n500<\/td>\n60%<\/td>\n<\/tr>\n
File5.cpp<\/td>\n500<\/td>\n60%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

oh wow \u2013 I\u2019ve barely tested the functionality in one of the files (File1.cpp). One step deeper confirms the story.<\/p>\n\n\n\n\n\n\n\n
File1.cpp Drill Down<\/td>\nCode Coverage<\/td>\n<\/tr>\n
SomeFunc<\/td>\n10%<\/td>\n<\/tr>\n
SomeOtherFunc<\/td>\n0%<\/td>\n<\/tr>\n
SomePeculiarFunc<\/td>\n0%<\/td>\n<\/tr>\n
ImportantFunc<\/td>\n3%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

From here, I\u2019d look at each of the functions in this file and see what testing I\u2019m missing that may help me hit this code. Once again, all I\u2019ve done is used the code coverage data to guide my testing. Once I\u2019ve looked at what I may be missing in File1.cpp, I may back out and look at Format Codes, or I may back all the way back to looking at Filters. I still use my knowledge of risk and priority to tell me where<\/em> to investigate, then use the coverage data as a tool<\/em> to help guide me in the right direction.<\/p>\n

An important thing to reiterate, is that I\u2019m still not concerned with improving the code coverage number<\/em> \u2013 I just want to use the data to see what I\u2019m missing. Sure, the number will go up anyway, but it\u2019s not the point. The point is reducing risk by identifying missing test areas. <\/p>\n","protected":false},"excerpt":{"rendered":"

I\u2019m going to do one more (I think) post on this subject. Markus asked the following about my latest post: But how do I answer the question for \u201cwhat\u2019s the coverage of your testing?\u201d for a multiple component-based application, consisting of a C\/C++ major component, an application server, and customized business logic? The answer is…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-176","post","type-post","status-publish","format-standard","hentry","category-allposts"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/176","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/comments?post=176"}],"version-history":[{"count":0,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/posts\/176\/revisions"}],"wp:attachment":[{"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/media?parent=176"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/categories?post=176"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angryweasel.com\/blog\/wp-json\/wp\/v2\/tags?post=176"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}