Testing: Failing to Succeed

…the paradox of testing: We want to find defects in the software under test, but we do not really want to find them".”Phillip Armour

I often speak of Phillip Amour’s Five Orders of Ignoranceas they relate to software engineering (note; as of this writing, that link doesn’t work, but I don’t have another link to his paper). I’ve mentioned the orders of ignorance on this blog, and also in this article for Better Software.

Although I respect my readers, I know you won’t read those links, so let me briefly cover the orders of ignorance.

0OI – Zero-Oh-I is lack of ignorance. It’s when you know something. I know, for example, that the first track on the Rolling Stone’s Sticky Fingers album is Brown Sugar.

1OI – lack of knowledge. I don’t know (or remember) what the track is following Brown Sugar, but I could find the answer quickly.

2OI – lack of awareness. You have 2OI when you don’t know what you don’t know. I know that there are Stones tunes that I’ve never heard before, but it would be impossible for me to make a list of Stones songs I’ve never heard before.

30I – lack of process. You have 3OI when you don’t have a suitable method for discovering 2OI (for discovering what you don’t know you don’t know).

4OI – meta-ignorance. You have 4OI when you don’t know about the five levels of ignorance.

In the Better Software article, I attempted to stress how important the levels of ignorance are to software testers and software testing, and I frequently bring them up in conversations about testing. Recently, my mind was blown when Phillip Armour wrote an article specifically about the application of the levels of ignorance to software testing. A direct link (for those who have access to the communications of the ACM) is here. If not, I found a link here which should get you to the article.

And for those of you who don’t click through, here’s a summary.

When we test to ensure requirements were implemented, or the stories function as expected, we are testing for 0OI. We know what pass and fail looks like, and we can create tests accordingly. The 0OI testing set may be large, but it’s bounded. A passing 0OI test doesn’t expose any new knowledge – it just proves what we thought we already knew.

Testers also perform a lot of 2OI testing. We explore or run dynamic tests on a system to help discover what we don’t know we don’t know. Testing for 2OI is an unbounded test set – the things that a system might do is infinite. This is where we need to devise tests when we don’t know what we’re looking for. (Note – we don’t test for 1OI because if we truly knew in advance what we didn’t’ know, we’d resolve the ignorance and conduct 0OI testing.

Another (shorter) way to think about this is that 0OI tests are knowledge-proving tests, while 2OI tests are knowledge-acquiring tests.

Armour goes on to discuss the theoretical information content for a test and optimal failure rates (which I’ll leave to you to read, as I’ve probably already butchered the content already). It’s good stuff full of great quotes, and I hope you check it out.