Putting the pieces together

My last two posts (one on Tester DNA, and the most recent on worries about the future) were written in isolation. The started the DNA post many months ago, and the future at least a year ago. I started and stopped completing the posts several times since their inception before finally completing them. For a bit of perspective, I currently have nine blog posts in my draft folder. Sometimes I think of something I want to write about, but realize that I don’t know what to say. When this happens, I start writing anyway – no matter what comes out. When I feel like I’ve written enough to dump my thoughts and save a draft. Then, I take a look at my drafts folder every week or so and see if there’s anything I want to finish – most just sit there until I give up and delete them. Overall, though, I would guess that a tenth or so of my posts, at most, sit in the draft folder before being posted – for blog posts, I favor much more of a stream of consciousness approach over heavy editing (which is easier for me, but I imagine it can be difficult for my readers).

Anyway…I ended up posting two random articles in a row from my drafts folder backlog. I didn’t mean for the posts to be related, but they are – I guess – or maybe I’m stretching things. At any rate, in response to my “the future is bleak” post Rikard Edgen asked which types of bugs I thought testers would miss. I thought I’d (kind of) answer that, and also explain why systems thinking is so important for testers.

Let’s start small – so small that we may not even need testers! Here’s a bug free program**.

void main()
{
    printf("Hello World");
}

Not very exciting – it doesn’t really do anything (**and it can’t be localized). Overall, it’s pretty useless, so for example’s sake, let’s expand our view of small to a small class or unit of functionality – you know, about the size you’d write unit tests for. You may have a bit of functionality – for example, selecting a book from a database. I like to picture a unit of functionality like this:

The unit is much bigger than the “hello world” example above, but with good design and good unit testing, it’s quite possible to write an entire unit that’s nearly bug free (from a functional level at least). In fact – I often speak of a dream of mine where testers never find unit level functional bugs, and that developer tools and approaches eradicate these types of errors in software forever…but I dream a lot.

People expect software do be useful, and an application that does nothing but select a book from a database wouldn’t be very useful at all– people probably want to add or read reviews, see publisher information, sort book lists, or add the book to a shopping cart for purchase. By the time the software ends up being usable (or marketable), it has a lot of units.

Now – even if all of the units are well designed and well unit tested, I would bet money and stock options that the application has plenty of bugs. This is because many bugs will occur where these units connect into larger pieces of functionality. In 1999, NASA lost $125 million because of a metric mismatch between two units (I bet the units functioned perfectly in isolation). These “connection points” (or dependencies) are a pretty common place for bugs to occur. The good esters understand where the pieces connect and “see the big picture” of the application in order to guide their testing. The testers with the systems thinking bit in their DNA, do pretty well at this, but it can be challenging for even the best of testers as applications get larger. At some point testers will need to use analysis tools to help them understand the connection points and dependencies, but their brain is still valuable in understanding the big picture of how the bits work together.

Now – as applications get bigger, this get’s harder. It’s impossible for a single tester to keep the big picture of something like SQL Server or Excel in their head – even a “trivial” application like the authoring client I’m using to write this post has a lot of moving parts and can be difficult to keep track of from a systems view.

One of my worries about the future is that trivial applications won’t exist anymore – ok – they’ll exist, but they’ll be part of much larger systems. In the examples above, those units that connected together into functional pieces became functional pieces that connect together into applications. Now, envision a world where applications (including services and devices) connect into mega-applications,services, and a world of interconnected deices.

If you believe my theory about bugs occurring at connection points, and that hugely interconnected software systems of the (near) future will look like this, you should realize that this is going to be just a bit more difficult to test. My stance is that you just can’t test a system such as this the same way you test a simple application. Sure, we’ll need to design the software so there are fewer connection points – or that the connection points are well managed and tested (you know – better than they did on the orbiter team). That also means that it would be beneficial if testers could test the design of the software (and that the design is testable). I expect testers will use sophisticated tools to help target their testing and recognize where error prone areas exist (and be able to run the right types of tests to reduce risk). An exploratory approach will still be prominent, but the challenge is where to look first, and what to do when you get there. This is where I think the tester DNA – problem solving, quick learning, and big picture thinking will be critical. And it will still be extremely difficult to test such a system.

Of course, it’s a valid argument to just build a system first then wait for test to catch up. That’s pretty much what’s occurred in software engineering up to today, and software is still advancing (although software quality, arguably, is not). My worry is that gigantic software systems are even more prone to critical errors due to an exponential number of code paths as well as the “death by a thousand cuts” effect of thousands of components – each containing “only a few” bugs.

My opinion is that we actually need to improve the way we do testing (actually, the way we create software from beginning to end) before we can pull off building a system like this. There’s a good chance I’m completely wrong…but maybe I’m right.