My last two posts (one on Tester DNA, and the most recent on worries about the future) were written in isolation. The started the DNA post many months ago, and the future at least a year ago. I started and stopped completing the posts several times since their inception before finally completing them. For a bit of perspective, I currently have nine blog posts in my draft folder. Sometimes I think of something I want to write about, but realize that I don’t know what to say. When this happens, I start writing anyway – no matter what comes out. When I feel like I’ve written enough to dump my thoughts and save a draft. Then, I take a look at my drafts folder every week or so and see if there’s anything I want to finish – most just sit there until I give up and delete them. Overall, though, I would guess that a tenth or so of my posts, at most, sit in the draft folder before being posted – for blog posts, I favor much more of a stream of consciousness approach over heavy editing (which is easier for me, but I imagine it can be difficult for my readers).

Anyway…I ended up posting two random articles in a row from my drafts folder backlog. I didn’t mean for the posts to be related, but they are – I guess – or maybe I’m stretching things. At any rate, in response to my “the future is bleak” post Rikard Edgen asked which types of bugs I thought testers would miss. I thought I’d (kind of) answer that, and also explain why systems thinking is so important for testers.

Let’s start small – so small that we may not even need testers! Here’s a bug free program**.

void main()
{
    printf("Hello World");
}

Not very exciting – it doesn’t really do anything (**and it can’t be localized). Overall, it’s pretty useless, so for example’s sake, let’s expand our view of small to a small class or unit of functionality – you know, about the size you’d write unit tests for. You may have a bit of functionality – for example, selecting a book from a database. I like to picture a unit of functionality like this:

The unit is much bigger than the “hello world” example above, but with good design and good unit testing, it’s quite possible to write an entire unit that’s nearly bug free (from a functional level at least). In fact – I often speak of a dream of mine where testers never find unit level functional bugs, and that developer tools and approaches eradicate these types of errors in software forever…but I dream a lot.

People expect software do be useful, and an application that does nothing but select a book from a database wouldn’t be very useful at all– people probably want to add or read reviews, see publisher information, sort book lists, or add the book to a shopping cart for purchase. By the time the software ends up being usable (or marketable), it has a lot of units.

Now – even if all of the units are well designed and well unit tested, I would bet money and stock options that the application has plenty of bugs. This is because many bugs will occur where these units connect into larger pieces of functionality. In 1999, NASA lost $125 million because of a metric mismatch between two units (I bet the units functioned perfectly in isolation). These “connection points” (or dependencies) are a pretty common place for bugs to occur. The good esters understand where the pieces connect and “see the big picture” of the application in order to guide their testing. The testers with the systems thinking bit in their DNA, do pretty well at this, but it can be challenging for even the best of testers as applications get larger. At some point testers will need to use analysis tools to help them understand the connection points and dependencies, but their brain is still valuable in understanding the big picture of how the bits work together.

Now – as applications get bigger, this get’s harder. It’s impossible for a single tester to keep the big picture of something like SQL Server or Excel in their head – even a “trivial” application like the authoring client I’m using to write this post has a lot of moving parts and can be difficult to keep track of from a systems view.

One of my worries about the future is that trivial applications won’t exist anymore – ok – they’ll exist, but they’ll be part of much larger systems. In the examples above, those units that connected together into functional pieces became functional pieces that connect together into applications. Now, envision a world where applications (including services and devices) connect into mega-applications,services, and a world of interconnected deices.

If you believe my theory about bugs occurring at connection points, and that hugely interconnected software systems of the (near) future will look like this, you should realize that this is going to be just a bit more difficult to test. My stance is that you just can’t test a system such as this the same way you test a simple application. Sure, we’ll need to design the software so there are fewer connection points – or that the connection points are well managed and tested (you know – better than they did on the orbiter team). That also means that it would be beneficial if testers could test the design of the software (and that the design is testable). I expect testers will use sophisticated tools to help target their testing and recognize where error prone areas exist (and be able to run the right types of tests to reduce risk). An exploratory approach will still be prominent, but the challenge is where to look first, and what to do when you get there. This is where I think the tester DNA – problem solving, quick learning, and big picture thinking will be critical. And it will still be extremely difficult to test such a system.

Of course, it’s a valid argument to just build a system first then wait for test to catch up. That’s pretty much what’s occurred in software engineering up to today, and software is still advancing (although software quality, arguably, is not). My worry is that gigantic software systems are even more prone to critical errors due to an exponential number of code paths as well as the “death by a thousand cuts” effect of thousands of components – each containing “only a few” bugs.

My opinion is that we actually need to improve the way we do testing (actually, the way we create software from beginning to end) before we can pull off building a system like this. There’s a good chance I’m completely wrong…but maybe I’m right.

I’ve been thinking about writing this post for years, but have had it floating around between my head and a draft for the last few months. I’ve held on to the post because even though it’s not my intent, I’m probably going to piss off someone who takes my points the wrong way or misinterprets what I say.I’m still not sure where I’m going with it, but I need to try to get these thoughts out sometime, and now is as good of a time as any.

I’m worried about the future of software testing. To be fair, I’m probably as much to blame as others, but I’m so bothered about this that I’m actually beginning to get a bit depressed. But I suppose I should explain myself before I rant any more – so let’s start with the current state of software testing.

But before that (and in the hope of diffusing the hard message a bit), let me say that I’m quite happy with what’s been happening in software testing recently. The “voice” of software testing is gaining momentum, and there are many active participants in discussing the basics of software testing so that the new crop of testers has a strong base of knowledge for the future. We have brilliant people emerging in the world of software testing, and I love the passion they have for the craft. It’s great to see so many testers online discussing their experiences and working hard to hone their skills. But we’re still talking about the same stuff testers were talking about ten years ago. The testing community is abuzz with discussions about basic tester skills, basic automation approaches and basic approaches to measurement. The discussions are lively, and people are learning – but they’re learning the same things testers were learning last year, the year before that, and the year before that. I like presenting and attending conferences like STAR and other similar conferences – but those conferences are targeted at new testers, and talk topics are dominated by the same things topics that have been around for years. I checked out the STAR conference proceedings from ten years ago (you can look too), and it looks remarkably similar to this years conference. In some cases, the arguments and points are more refined than ten years ago, but a remarkable number of those talks could just as easily show up on any conference program this year.

The state of the art in software testing will go nowhere if we never look beyond the basics. Highlighting flaws in simple applications written by the IT department of a small company (or parking garage) is interesting, and it develops basic skills, but it does nothing to help us understand better ways to test huge interconnected systems or services under constant load. The future software systems shown in many science fiction movies may never come to light – and it won’t be because of technology or cost factors – it will be because we won’t know how to test these systems adequately.

One way to view the current body of knowledge in software testing something like the triangle below. At the bottom of the triangle is the “101” level of software testing. This is important stuff. It’s the basis of the skills, approaches, and techniques that are the core of software testing. My triangle is disproportionate, as his is probably where 99% of the software testers spend their time (ok – to be fair, some never even make it on to the triangle). The big problem here, as I said above, is that this enables the advancement of our future, but does little to nothing to help the future.

The middle section of the triangle – new ideas and approaches do occur semi-frequently, but we more often than not view them as game-changers when the impact on the future is relatively minimal. Like the bottom section of the triangle, we need  thinking in this area, but it’s not going to take us where we need to go. The top section of the triangle contains the true game changers. The new thoughts (or more likely, thoughts and ideas from some other area applied to testing) that will bring us into a new era.

Let me return to my worry (and a problem I mentioned above). My fear is this: The software testing we are doing today will be inadequate for the software of tomorrow. It doesn’t matter how good we are at the basic skills, it won’t be enough. What I fear is going to happen is that we will go ahead and create the software of tomorrow but we’ll try to test it like we test software today. For example’s sake, envision the systems of “Minority Report” – most of the pieces of technology in this movie have been demonstrated, but never with the flawless interconnection that Hollywood showed us, Do we have the skills and tools (and people) to test such a system today? Not. Even. Close.

What I’m afraid will happen, is that we’ll eventually build a similar system, and we will try to test it with the tools, approaches, processes (and many of the people) who are testing software today.

And there will be massive software failure, and very bad things will happen. At least then we may find enough motivation to advance the state of the art in testing.

But we don’t have to wait. If you’ve take a reasonably close look at the software of today you know that we could use some innovation and advancement now. The right thing to do is to start moving beyond the basics and figure out how we can do better testing today. How do we test massively complex systems? How do we do so efficiently? Are the software testers of today the right people to test the software of the future?

Moving up the triangle is a hard move to make. Testing consultants make their money in the bottom of the triangle, so why would they have any motivation to make a move outside of the cash flow? But it’s not the consultant’s fault either. The testing profession has a high enough turnover that there are a huge number of new testers every year looking to get their foothold in the bottom rung of that triangle. Also consider that most of the testers today aren’t testing huge systems used by millions of users – they’re testing one-off IT apps used by hundreds (or dozens) of people who don’t care if the software has a few minor issues. There’s no need (or time) for most of these testers to ever get out of the bottom rung of the triangle.

It’s vicious circle, but we have to find a way out.The future depends on it.

I wish I could say that my next blog post will have all the answers. (it won’t). Instead, I’m going to continue to study, learn, and experiment. Somewhere out there is the knowledge that we’ll need to survive the software of the future, and I want to be part of making that future software successful. I hope that some of you can help make it happen as well.