Archive for the ‘Quality’ category

It must be simple

May 31st, 2010

Many of you know that I’m a big fan of the five orders of ignorance (link is to my interpretation – the author’s original paper is here). I especially like the concept of “unknown unknowns” those bits of knowledge that are critical – despite the fact that we didn’t recognize those bits until they fell onto our laps.

I’ve been pondering over an interesting fuzzy line between Armour’s second level of ignorance (you know you don’t know something), and his third level of ignorance (you don’t know you don’t know). I’ve noticed that often, when you know nothing about a subject, you perceive it as simple. Then, as you learn more, it seems impossibly difficult; and then, as you master it, it become’s easier to manage (but probably never as simple as you imagined in the first place). The second and third levels of ignorance seem to play with each other as you try to move knowledge into the first level of ignorance (you know what you know)

An example would probably help here. Many years ago, I knew nothing about recording music, but I wanted to be able to record my own stuff, overdub, mix, edit, etc. Since I had never done it before, I was convinced it was easy, and that anyone with a brain and a halfway working set of ears should be able to do it.Within a few weeks of attempting to pick up this “easy task”, I realized it was extremely complicated and required a lot of meticulous work (this was pre-digital, so within a few weeks I was fixing timing issues with a razor blade). I didn’t think I’d ever get a handle on it, but eventually I figured enough stuff out for my own purposes, and to this day I still uncover areas of complexity I never imagined.

There’s another great example of this from my list of favorite movies that came out when I was 19 – Better off Dead. There’s a scene where John Cusack’s character is on top of a mountain ready to ski down and his friend gives him this advice:

Go that way, really fast. If something gets in your way, turn

That’s exactly the “expert” advice someone who doesn’t know the subject would give. Steve Martin used to tell a similar joke about skiing – I don’t remember it exactly, but it went something like this:

Skiing – you go to the top of a mountain with slippery things on each foot and go down. Try not to – that would be a sport!

Perhaps a better example for testers is Jerry Weinberg’s Perfect Software book. To me, it’s not so much a testing book, but it’s a wonderful book to give to people who don’t know anything about testing – mainly because most people who don’t know about testing often think it’s a simple activity that has something to do with banging on keys or pushing buttons. Once you learn a bit about software testing, you of course realize that it’s far more complex than you ever could imagine. Eventually (if you’re lucky), you fall in love with the challenge and never look back.

I see at least a few examples of this phenomenon every day. I have lost track of the times I’ve heard people critiquing the efforts to plug the oil leak in the Gulf and offering their own advice. However, the fact that they haven’t been able to solve it, and have tried to fix it several ways must mean it’s probably a more complex problem than most of us understand (or that the efforts are run by aliens from the planet Stupid).

I suppose the moral of all this is that nothing is as simple as it appears – especially with knowledge acquisition. Keep it in mind the next time you think you know something …

Don’t forget the checklist

May 6th, 2010

I recently read The Checklist Manifesto: How To Get Things Right by Atul Gawande. This is yet another book not about testing that testers should read. Gawande is a surgeon and relates his experiences from the field of medicine throughout the book. )ne of the first chapters goes into depth on the growing complexity and body of knowledge in medicine, and how difficult it is to keep many of the important details of the practice in mind consistently. Sound Familiar?

In Gawande’s experiments, he looked at areas where there were “bugs” (“bugs” in his case were complications or death), and created short, simple checklists that could be used by the medical staff to reduce risk – and the checklists worked wonderfully – fore example, reducing infection in central line procedures by over 60%. He also points out the cause of error (I’m a huge fan of Reason’s Human Error) usually breaks down to errors of ignorance (not knowing the right thing to do), and errors of ineptitude (not knowing how to apply what we know).

Checklists aren’t new to testing. Kaner talks about checklists, and testers at Microsoft have been using “am I done” type checklists for at least 15 years (Michael Hunter has a great list here). Checklists are good tools for testers to use and probably worth looking at more closely.

But building a checklist is probably harder than you think. In my experience (Gawande saw the same thing), most checklists are too long and contain too much detail. They get in the way! To make matters worse, as you learn more, you will likely add additional items to your checklist. The solution is simple enough – start with the smallest possible checklist you think you can get away with (and then see if there’s anything else you can remove). Realize that your checklist can change over time as you learn more about your processes and the types of errors your team makes. Try it – I think you’ll find that checklists are a great help.

 

Remember -  don’t forget to use a checklist (you can always use another checklist as a reminder)

Books I’m about to (re)read

February 18th, 2010

I’m heading up to the Vancouver for the Olympics this weekend and hoping to catch up on some reading on the trip. I’ve built up a bit of a backlog of books I want to read or re-read and I’m going to try to make a big dent in the list during the train ride (and while waiting in line for will-call tickets).

First on the list is Saul Alinsky’s Rules for Radicals. I’ve been meaning to read it for a while and I’m looking forward to seeing what people love (and hate) about it. second book on the list is The Checklist Manifesto. Atul Gawande spoke on MS campus recently and his talk inspired me to take a closer look at his book.

There are two other books on my re-read list. Both are controversial in that people seem to either really, really love them, or really, really loathe them. The books couldn’t be more different, but the value in both of them to me is beyond the printed word – the value is between the lines, in the thoughts and ideas they bring out in me every time I read them. ZATAOMM is a classic and a book that I read every few years, or every time I change jobs (whichever comes first). I’m  also going to re-read Quality is Free. A lot of software folks hate this book. At face value, of course, application of manufacturing approaches to software often fail, but the point (to me at least) is to remind me that quality can be improved through processes and culture change and it always gets me thinking of new ideas for improving software quality and building a quality culture.

That’s probably more than enough reading for a short trip, but I’ll bring at least three of those with me just in case there’s more waiting than I anticipate. I think Rules for Radicals is the only book that’s not commonly read among my peers, so I’ll plan to share any interesting insights in a mini book report once I finish it.

Another Take on the Five Orders of Ignorance

February 14th, 2010

 

I came across a post this weekend about different types of knowledge It’s a well written post that calls out one of my favorite topics – you don’t know what you don’t know. I think the Five Orders of Ignorance (or my tribute to the five orders here) are brilliant, and the author nails Armour’s zero, first, and second level ignorance with his take on knowledge (described as “shit you know, shit you don’t know, and shit you don’t know you don’t know). 20I (read two-oh-eye) – what you don’t know you don’t know (aka the unknown unknowns) are a big reason why we suck at scheduling. Think about it – when we throw out estimates for how long a particular task will take, it’s often half guess, and half padding because we know “stuff” will come up (yes, there’s a lot we can do to make scheduling more accurate too, but that’s a topic for another day).

IMO, there are two pretty big things missing from this article. The first is a discussion of suitable methods to determine what we don’t know we don’t know (aka Armour’s 30I). A passion for learning, critical thinking (and the knowledge or acknowledgement of 20I) all help us identify and solve the unknown unknowns. The way we learn is to discover what we don’t know we don’t know, make it something we (just) don’t know, then make it something we know. We move knowledge down through the orders of ignorance – without a method for discovering what we don’t know we don’t know, it’s just a problem that we’ll never solve (or never know about). For some, I suppose that’s fine, but not for people who are interested in learning.

The second point is that the author doesn’t acknowledge Armour. What’s great about this is that I don’t think it was on purpose – I’m pretty sure the author just didn’t know that Armour had already discussed the same topic in depth. The unknown unknown’s will get you every time!

Listening to feedback

January 27th, 2010

I have half a dozen different blog posts started but I thought I’d write this one instead. When you listen to customer feedback, you need to treat the good and the bad equally. If some customers say “wow – this is awesome”, and some others say “this is awful – are you on crack?”, either they’re both right, or neither of them are right. People base their opinion on their experience. It’s unfortunate that you created polarizing experiences, but that’s what happened. If  you write off the negative comments as flukes, it’s only fair that you say the same about the positive comments.

We get feedback on the courses we teach for testers at Microsoft. My team is pretty good at reviewing feedback and making adjustments as necessary. I remind my team of the same thing. They need to treat the positive comments (“the instructor made the class awesome!”) and the negative comments (“the instructor was an idiot”) the same. Either throw them both out, or treat them both as valid. Obviously, you can play the balance game. If 20 students said “great”, and 4 said “sucks”, then you’re probably on the right track (although you’d want to self-evaluate on why the polarity exists in the first place). It’s a frequent topic of discussion, but one that I think everyone understands.

The concept came up again this week when a teammate pointed out that someone commented about me on the mini-microsoft blog. The comment in question was mostly positive, yet it was in response to something negative (which, in full disclosure, I replied to a week or so ago). In this case, I either have stale ideas and have some talent, or neither. I don’t really care either way, but I suppose I’m happy for the attention (there’s no such thing as bad publicity).

The important lesson to learn is that there’s no bad feedback, and no irrelevant feedback. There’s just feedback – it’s your choice to listen to it or not.

Settling on Quality?

November 8th, 2009

Oh my – another quality post. I’m afraid I’m starting a trend for myself, but I have a story to share.

As all gainfully employed workers in the tech field will tell you, we all have side jobs as tech support for all members of our immediate and extended families. This weekend, my mother-in-law opened a support ticket with me regarding her laptop – it was crashing randomly (that’s all the details you get when your m-i-l opens a support ticket).

So – I turned on her laptop, let it boot, then dealt with message after message from applications starting up and telling me stuff I didn’t care about. A backup program telling me that it needed a product key, an external hard drive utility telling me the drive wasn’t connected (duh), and an OEM replacement for windows wireless config launching to tell me I’m connected to a wireless network. The experience was annoying. But there’s a bigger problem. As I was looking at the 3 different web browsers installed and the few dozen or so other random programs and utilities installed, my first thought was “no wonder she’s having computer problems – she’s installed every app under the sun”. I always try to keep my main work machines somewhat “clean” – only installing applications I consider tried and true for worry that they’ll mess something up. Then I realized that’s wrong – I should be able to install whatever the hell I want without fear of losing overall quality (who knows – maybe I can and it’s all a mental problem on my end). The point is, that we (computer users) don’t seem to expect software to work. We’re not as surprised, alarmed, or pissed off as we should be when software doesn’t work correctly. Honestly – I’ve belittled people in the past for calling things bugs when they’re 99.99% user error, but I was wrong – user error or not, that .01% matters.

Ok, so software sucks. It really doesn’t matter – it’s still a profitable industry. That’s true, but I wonder how long it will be true. I wonder if something horrible (even worse than Windows ME** :}) has to happen before the world demands higher quality software. My hope is that we can start making better software long before something like that happens.

Oh – as far as my mother-in-laws computer goes, there was a crash dump on the machine. I attached a debugger and poked around a crash in the wireless driver. I put a later rev of the driver on the machine and so far, so good. I hope it stays that way…for at least a little while.

Finding Quality

November 5th, 2009

Leave it to Adam Goucher to beat me to the punch line. When I proposed that breaking down your definition of quality to a manageable set of ilities is a reasonable method for improving customer perceived quality, the logical next step is to try and find out which of the ilities you need to care about. Adam suggestion was:

Want to improve v2? Talk to customers of v1 and ask which of the ilities they suffer the most from. And/or talk to people who didn’t buy your software and ask them which ility chased them away. Those are the ones that count.

Perfect answer, but keep in mind that the questions you ask are critical. You can’t ask “do you want the product to be more reliable”, or even “from this list, choose the one you care about the most”. For former question will always result in a “yes” answer, and the second will probably just result in confusion. Instead, you can ask questions like “tell me what you like most about the software (or what you dislike the most”. Ask open ended questions and take notes – take lot’s of notes. After  you’ve talked to a good sample of customers, break the notes into individual comments and start sticking them on a wall. Look for affinity – start grouping items and looking for themes. Then, see if an ility aligns with a theme. Eventually, you’ll have a bunch of big fat quality bulls-eyes on the wall waiting for you to address.

I have one minor nit where Adam missed the mark – you don’t have to wait until v1 is out to collect this data. If your software team is worth their salt, they’ve defined the customer segments they care about far before v1 hits the street. Interview customers from that segment and ask them questions like “this product does foo, what do you expect a high quality product that does foo to do?”, or “what would make you want to use a product like this?” or “what would make a product that does foo unusable for you?”

The fun part is that I’ve sort of done this (in a very general way), and have a list (that certainly won’t work for every piece of software in the world, but is worth discussing). I haven’t yet figured out how to push the dial on these ilities, but but that’s what I’m going to try and figure out – using this blog as a sounding board while I think.