Musings on Test Design

The wikipedia entry on test automation troubles me. It says:

Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.

I can’t decide whether it bothers me because it’s wrong, or because so many people believe the statement is true. In fact, I know the approach in some organizations is to “design” a huge list of manual tests based on written and non-written requirements, and then either try to automate those tests, or pass the tests off to another team (or company) to automate.

This is an awful approach to test design. It’s not holistic. It’s short-sighted. It’s immature, and it’s unprofessional. It’s flat-out crummy testing. I frequently say, “You should automate 100% of the tests that should be automated”. Let me put it another way to be clear:

Some tests can only be run via some sort of test automation.
Some tests can only be done via human interaction.

That part (should be) obvious. Here’s the part I don’t think many people get:

You can’t effectively think about automated testing separately from human testing.

Test Design answers the question, “How are we going to test this?” The answer to that question will help you decide where automation can help (and where it won’t).

Here’s a screen shot of part of the registration form for outlook.com (disclaimer – I have no idea how this was tested).

Let’s look at two different ways of answering the “How will we test this?” question.

The “automator” may look at this and think the following.

• I’ll build a table of first and last names and use those for test accounts
• I’ll try every combination of Days, Months, and Years for Birthdate
• I’ll generate a bunch of different test account names
• I’ll create a password for each account
• Once I try all of the combinations, I can sign off on this form
• (or, they may think, “I wonder what sort of test script the test team will ask me to automate”

The “human” may look at the same form and think this:

• I’ll want to try short names, long names, and blank names
• I’ll see if I can find invalid dates (e.g. Feb 29 in a non-leap year)
• Some characters are invalid for email names – I’ll try to find some of those
• I’ll make sure the 8-character minimum and case sensitivity is enforced
• Oh – I’ll try that passwords with foreign characters too.
• Once I go through all of that, and anything else I discover, I can sign off on this form.

I’ll fire off a disclaimer now, because I’ve probably pissed off both “automators”, and “humans” with the generalizations above. I know there’s overlap. My argument is that there should be more.

In my contrived examples above, the “automator” is answering the question, “What can I automate?”, and the “human is answering the question, “What can I explore or discover?”. Neither is directly answering the question, “how are we going to test this?”.

I could just merge the lists, but that’s not exactly right. Let’s throw away humans and coders for a minute and see if we can use a mind map to get the whole picture together. Here’s what I came up with giving myself a limit of 10 minutes. There’s likely something missing, but it’s a starting point.

Now (and at no time before now), I can begin to think about where automation may help me. My goal isn’t to automate everything, it’s to automate what makes the most sense to automate. Things like submitting the form from multiple machines at once, or cycling through tables of data make sense to automate early. Other items, like checking for non-existent days or checking max length of a name are nice exploratory tasks. And then, there are ideas like foreign characters in the name, or trying RTL languages that may start as an exploratory test, but may lead to ideas for automation.

The point is, and this is worth repeating so often, is that thinking of test automation and “human” testing as separate activities is one of the biggest sins I see in software testing today. It’s not only ineffective, but I think this sort of thinking is a detriment to the advancement of software testing.

In my world, there are no such things as automated testing, exploratory testing, manual testing, etc.

There is only testing.

More musings and rants to follow.