If you ever want to start a great discussion / debate with a group of testers, ask them to define quality and come up with measurements.“Quality” is such an overloaded term that it’s hard to get people to agree. The Weinberg fans (I suppose I’m one of them) will cite their mentor and say “Quality is Value (to some person)”. To that, I say fine – define Value and tell me how to measure that! Most of the time, I think measuring quality is similar to how Potter Stewart defined pornography when he said “it’s hard to define, but I know it when I see it”. I’ll admit that in many situations, the value of gut-feel and hunches about quality outweigh some of the quantitative attempts some organizations use. Unfortunately, I see many quality “experts” throw the baby out with the bathwater and dismiss quantitative metrics simply because they’re easy to get wrong.
If you ask most teams how they measure quality, they’ll probably tell you they measure code coverage, count test cases, track bugs found during different activities and a number of other engineering practices. They’re not improving product quality – they’re improving engineering quality. Now, improving engineering quality is a good thing too – it does a lot to decrease risk and increase confidence, but it does diddly-squat to improve the customers perception of quality. So here’s a conundrum – how do you measure perception before you can get someone to perceive it? One way is to define scenarios, and attempt to use the software like we think customers will use it, all the while noting where it’s difficult, or where something goes wrong, then working to get those issues fixed. In the end, we still cross our fingers and hope they’ll like what we gave them.
But I’m wondering if there’s a better way to make software our customers like (and perceive to be of high quality). Wikipedia has a great list of the ilities – attributes that lead to system quality, but the list is huge. If you attempt to improve that many items at once, you may as well work on nothing at all. But suppose you knew which of those ilities were most important to your most important customer segments. My hypothesis is that if you focus on pushing the bar on a small set of quality attributes that customer perception of quality will improve. It’s not easy (again – why some people just give up completely on metrics), but I think it can work.
Think of this scenario: You’re leading a team on v2 of their software. You ask them to “improve quality”. If their heads don’t spin and explode, they may ask to hire more testers or make plans to increase code coverage, or they may just ignore you because quality is too hard to measure. Or, you could tell your team they need to focus on reliability, usability, and compatibility (areas you know your customers care the most deeply about). You provide them with checklists and other aids to help them think about these areas more deeply as they work on v2. You may even come up with some measurements or models that show how much the team is improving in those areas.I’m fairly confident one of those approaches will lead to quality software 99 times out of 100.
I’ll dig into some of my favorite ilities and speculate how to improve them in future posts.
Ed: added the rest of the Weinberg quote because so many people were annoyed I left it out.
What about measuring testers? Measuring quality is as you stated a rat hole.
Is trying to measure a tester, like a baseball player, or football player is measured a similar rat hole or worth talking about?
What is the parallel to HRs, RBIs, and batting average in testing? Would those stats be of value to measure testers, and then maybe that could lead us to the ladder that gets out of the how do we measure quality rat hole?
Alan,
So you seem to distinguish between two kinds of quality – engineering quality and product quality.
Is product quality an attribute intrinsic to whatever you call as “product” ? Something that is built into it?
A better differentiation is engineering quality (quality of the engineering process and methods), and quality of the customer experience (does the product satisfy my needs – both on a functional and emotional level)
Is engineering quality same as process quality?
I suppose – but I’ll stick with my definition above in case you have a different definition for process quality
Can a good (in whatever sense) engineering process produce a product with bad product quality?
ABSOLUTELY Yes – this is a big problem today
and conversly a good product quality an indication of good engineering quality?
not at all for the same reasons
Also what about interplay between these two “qualities in a social, cultural, economic and business context of software production and usage?
I think you have to worry about both in conjunction. A mistake I’m starting to see is software teams relying too heavily on either one of these without striving for a blend. The two overlap and feed off each other to some extent, so it’s important to think holistically about quality if you are really serious about it.
Shrini
Reid –
I think measuring individuals in testing is tough. The value in top testers isn’t in the stats – it’s in the intangibles. The reason Ken Griffey was on the Mariners last year wasn’t for his average or home run count. He was an experienced player who helped bring the team together – that’s the kind of stuff that “super” testers do. Focusing on the stats finds the wrong kind of thing more often than not.
If you want a metric, I’ve always thought that the best testers are the testers who have made everyone around them better. I’d want a tester whose baseball card said:
2009 Ratings
Peer Tester rating 9/10
Peer Dev Rating 8/10
Manager Rating 10/10
Group Rating 9/10
A tester like this is well respected among their testing peers, works well with development, has a great relationship with their manager, and is 9 out of 10 among their entire group. Sounds to me like a good tester. I don’t know their skills from this, but I don’t care a whole lot – these numbers tell me that the tester works well with others and earns credibility and respect from their team. That’s good stuff.
You are missing the second half of the Value definition which is ‘to some person that matters’ (my paraphrasing). Want to improve v2? Talk to customers of v1 and ask which of the ilities they suffer the most from. And/or talk to people who didn’t buy your software and ask them which ility chased them away. Those are the ones that count.
Alan: On the second half – you are one step ahead of me and exactly correct.
As for the first comment, I didn’t see the “to some person” part as adding to, or taking away from the statement, so I left it out. Kaner (I believe) added the “who matters” part.