Beyond Regression Tests

In a recent talk on test design (link), I discussed the concept of "useful tests". In my definition, useful tests are tests that provide new information. Almost every test is useful…once – typically the first time it’s run where it shows that the underlying functionality is working. From that point on, many tests function primarily as regression tests. We run these tests as a safeguard to ensure that we don’t break anything with future changes. They’re great for this purpose – and when they find a bug, they are wonderfully useful. But 99% of the time (according to my definition) they are not useful – they don’t provide new information (they do provide information – the information that nothing has changed, but it’s not new information).

To be fair, these are exactly the types of tests I want for unit tests (and probably acceptance tests too, depending on the definition). I love using unit tests to give everyone on the team confidence that refactoring or feature additions do not break basic functionality – but unit tests (and acceptance tests) are usually just a small portion of the overall testing effort.

I should stop here to clarify. On some teams who "do automation", the unit and acceptance test automation is all they do. In this case, the regression only focus may be appropriate.

Fortunately, the teams I work on (and with) use automation for far more than regression testing – , we need tests that can be "useful" (provide new information) way more frequently than when something that was working once stops working. Test automation isn’t just for regression testing It cannot (and shouldn’t try to) replace human brain-engaged testing, but it can certainly do more than automate a bunch of rote tasks. In my opinion, an automation strategy that only performs regression testing is short-sighted and incomplete.

So – how do you write an automated test that provides new information more frequently? A big advantage of human testing is what I call the "I wonder…" principle. For example, "I wonder what will happen if I cancel this operation in the middle?" or "I wonder if this will work if I pull out the network cable?". Computer programs aren’t very good at wondering, but they are good at brute force. Model-based testing is a potential solution here, as you can use it to traverse decision points in the application (randomly, or by applying a graph traversal technique). Traversing an application in unexpected ways often results in finding new issues. Same test, different bugs – and that seems pretty useful to me. Some testers, for whatever reason, seem to be a bit afraid or skeptical of MBT. I think in many cases, potential adopters of MBT either try to do too much with it before they understand the concepts, or think that it’s a replacement for some other types of testing. MBT is just another test design technique – and like other test design techniques, it works great in particular situations, and not so well in others.

But MBT isn’t the only way to get tests to provide different information on subsequent runs. Simply introducing some randomness to a test can often turn up new issues. For example if you have a "suite" of tests that always run in the same order, try running them in a random order on each run (note – log the random seed so you can reproduce any discovered issues later). Or, say you have a test that does a particular operation five times. Instead of running the operation five times on every test, what about running the operation [between 1 & 10 times] randomly. Regression tests, for all the value they bring, tend to train the system to pass. By mixing things up, we often find new issues.

I’m also a fan of data driven testing. Get the test data out of the automation code and into a data file (use XML or another DSL as appropriate). Then, randomize the data used by the test (and automate the randomization too). Then when you’re testing a web page where users enter address information, you can easily mix up a variety of name, address, zip code, etc. data (especially including invalid data), and see what happens. If you send the same data to the system every time, it will certainly pass all of your tests eventually, but I’m almost certain that you’ll still miss issues too.

Depending on what you’re testing, I’m sure I can think of other ideas that may make your automated tests useful more often. There’s nothing wrong with regression testing, but if that comprises all of your test automation effort, you’re probably leaving some cards (and bugs) on the table.