Anyway…I ended up posting two random articles in a row from my drafts folder backlog. I didn’t mean for the posts to be related, but they are – I guess – or maybe I’m stretching things. At any rate, in response to my “the future is bleak” post Rikard Edgen asked which types of bugs I thought testers would miss. I thought I’d (kind of) answer that, and also explain why systems thinking is so important for testers.

Let’s start small – so small that we may not even need testers! Here’s a bug free program**.

void main()
{
    printf("Hello World");
}

Not very exciting – it doesn’t really do anything (**and it can’t be localized). Overall, it’s pretty useless, so for example’s sake, let’s expand our view of small to a small class or unit of functionality – you know, about the size you’d write unit tests for. You may have a bit of functionality – for example, selecting a book from a database. I like to picture a unit of functionality like this:

The unit is much bigger than the “hello world” example above, but with good design and good unit testing, it’s quite possible to write an entire unit that’s nearly bug free (from a functional level at least). In fact – I often speak of a dream of mine where testers never find unit level functional bugs, and that developer tools and approaches eradicate these types of errors in software forever…but I dream a lot.

People expect software do be useful, and an application that does nothing but select a book from a database wouldn’t be very useful at all– people probably want to add or read reviews, see publisher information, sort book lists, or add the book to a shopping cart for purchase. By the time the software ends up being usable (or marketable), it has a lot of units.

Now – even if all of the units are well designed and well unit tested, I would bet money and stock options that the application has plenty of bugs. This is because many bugs will occur where these units connect into larger pieces of functionality. In 1999, NASA lost $125 million because of a metric mismatch between two units (I bet the units functioned perfectly in isolation). These “connection points” (or dependencies) are a pretty common place for bugs to occur. The good esters understand where the pieces connect and “see the big picture” of the application in order to guide their testing. The testers with the systems thinking bit in their DNA, do pretty well at this, but it can be challenging for even the best of testers as applications get larger. At some point testers will need to use analysis tools to help them understand the connection points and dependencies, but their brain is still valuable in understanding the big picture of how the bits work together.

Now – as applications get bigger, this get’s harder. It’s impossible for a single tester to keep the big picture of something like SQL Server or Excel in their head – even a “trivial” application like the authoring client I’m using to write this post has a lot of moving parts and can be difficult to keep track of from a systems view.

One of my worries about the future is that trivial applications won’t exist anymore – ok – they’ll exist, but they’ll be part of much larger systems. In the examples above, those units that connected together into functional pieces became functional pieces that connect together into applications. Now, envision a world where applications (including services and devices) connect into mega-applications,services, and a world of interconnected deices.

If you believe my theory about bugs occurring at connection points, and that hugely interconnected software systems of the (near) future will look like this, you should realize that this is going to be just a bit more difficult to test. My stance is that you just can’t test a system such as this the same way you test a simple application. Sure, we’ll need to design the software so there are fewer connection points – or that the connection points are well managed and tested (you know – better than they did on the orbiter team). That also means that it would be beneficial if testers could test the design of the software (and that the design is testable). I expect testers will use sophisticated tools to help target their testing and recognize where error prone areas exist (and be able to run the right types of tests to reduce risk). An exploratory approach will still be prominent, but the challenge is where to look first, and what to do when you get there. This is where I think the tester DNA – problem solving, quick learning, and big picture thinking will be critical. And it will still be extremely difficult to test such a system.

Of course, it’s a valid argument to just build a system first then wait for test to catch up. That’s pretty much what’s occurred in software engineering up to today, and software is still advancing (although software quality, arguably, is not). My worry is that gigantic software systems are even more prone to critical errors due to an exponential number of code paths as well as the “death by a thousand cuts” effect of thousands of components – each containing “only a few” bugs.

My opinion is that we actually need to improve the way we do testing (actually, the way we create software from beginning to end) before we can pull off building a system like this. There’s a good chance I’m completely wrong…but maybe I’m right.

I’m worried about the future of software testing. To be fair, I’m probably as much to blame as others, but I’m so bothered about this that I’m actually beginning to get a bit depressed. But I suppose I should explain myself before I rant any more – so let’s start with the current state of software testing.

But before that (and in the hope of diffusing the hard message a bit), let me say that I’m quite happy with what’s been happening in software testing recently. The “voice” of software testing is gaining momentum, and there are many active participants in discussing the basics of software testing so that the new crop of testers has a strong base of knowledge for the future. We have brilliant people emerging in the world of software testing, and I love the passion they have for the craft. It’s great to see so many testers online discussing their experiences and working hard to hone their skills. But we’re still talking about the same stuff testers were talking about ten years ago. The testing community is abuzz with discussions about basic tester skills, basic automation approaches and basic approaches to measurement. The discussions are lively, and people are learning – but they’re learning the same things testers were learning last year, the year before that, and the year before that. I like presenting and attending conferences like STAR and other similar conferences – but those conferences are targeted at new testers, and talk topics are dominated by the same things topics that have been around for years. I checked out the STAR conference proceedings from ten years ago (you can look too), and it looks remarkably similar to this years conference. In some cases, the arguments and points are more refined than ten years ago, but a remarkable number of those talks could just as easily show up on any conference program this year.

The state of the art in software testing will go nowhere if we never look beyond the basics. Highlighting flaws in simple applications written by the IT department of a small company (or parking garage) is interesting, and it develops basic skills, but it does nothing to help us understand better ways to test huge interconnected systems or services under constant load. The future software systems shown in many science fiction movies may never come to light – and it won’t be because of technology or cost factors – it will be because we won’t know how to test these systems adequately.

One way to view the current body of knowledge in software testing something like the triangle below. At the bottom of the triangle is the “101” level of software testing. This is important stuff. It’s the basis of the skills, approaches, and techniques that are the core of software testing. My triangle is disproportionate, as his is probably where 99% of the software testers spend their time (ok – to be fair, some never even make it on to the triangle). The big problem here, as I said above, is that this enables the advancement of our future, but does little to nothing to help the future.

The middle section of the triangle – new ideas and approaches do occur semi-frequently, but we more often than not view them as game-changers when the impact on the future is relatively minimal. Like the bottom section of the triangle, we need  thinking in this area, but it’s not going to take us where we need to go. The top section of the triangle contains the true game changers. The new thoughts (or more likely, thoughts and ideas from some other area applied to testing) that will bring us into a new era.

Let me return to my worry (and a problem I mentioned above). My fear is this: The software testing we are doing today will be inadequate for the software of tomorrow. It doesn’t matter how good we are at the basic skills, it won’t be enough. What I fear is going to happen is that we will go ahead and create the software of tomorrow but we’ll try to test it like we test software today. For example’s sake, envision the systems of “Minority Report” – most of the pieces of technology in this movie have been demonstrated, but never with the flawless interconnection that Hollywood showed us, Do we have the skills and tools (and people) to test such a system today? Not. Even. Close.

What I’m afraid will happen, is that we’ll eventually build a similar system, and we will try to test it with the tools, approaches, processes (and many of the people) who are testing software today.

And there will be massive software failure, and very bad things will happen. At least then we may find enough motivation to advance the state of the art in testing.

But we don’t have to wait. If you’ve take a reasonably close look at the software of today you know that we could use some innovation and advancement now. The right thing to do is to start moving beyond the basics and figure out how we can do better testing today. How do we test massively complex systems? How do we do so efficiently? Are the software testers of today the right people to test the software of the future?

Moving up the triangle is a hard move to make. Testing consultants make their money in the bottom of the triangle, so why would they have any motivation to make a move outside of the cash flow? But it’s not the consultant’s fault either. The testing profession has a high enough turnover that there are a huge number of new testers every year looking to get their foothold in the bottom rung of that triangle. Also consider that most of the testers today aren’t testing huge systems used by millions of users – they’re testing one-off IT apps used by hundreds (or dozens) of people who don’t care if the software has a few minor issues. There’s no need (or time) for most of these testers to ever get out of the bottom rung of the triangle.

It’s vicious circle, but we have to find a way out.The future depends on it.

I wish I could say that my next blog post will have all the answers. (it won’t). Instead, I’m going to continue to study, learn, and experiment. Somewhere out there is the knowledge that we’ll need to survive the software of the future, and I want to be part of making that future software successful. I hope that some of you can help make it happen as well.

Testing is broad and evolving – nobody knows everything about it, so the ability to learn quickly is critical (this aids in problem solving as well). If you’re the type that takes a long time to ramp up in new technologies or concepts, you may struggle as a tester. Probably more critical is a passion for learning. Good testers don’t wait for ideas to come to them, instead, they seek out knowledge  – they not only learn what they know they need to learn, they find ways to learn what they don’t know (i.e. they strive to resolve second level ignorance). I believe that the big innovations in testing will come from applying knowledge from outside the field of software and software testing. In order to advance the state of the art in testing, we need testers who seek knowledge – and who are able to apply those abstract concepts to solve some of our big problems in test.

But – to take care of that last sentence, you’re going to need people who can see the big picture – systems thinkers. There are numerous people who claim to be systems thinkers, but systems thinking takes practice as well as some innate ability (or DNA) to be beneficial – and it’s much harder than many people think. Often when I interview testers, I ask a “testing” question that has two parts to solve. The first (the question I actually ask them) is obvious and has a solution that is difficult enough that they solve it as they would any other question. However – there’s a hidden problem in the question. The good testers quickly see the secondary problem as the far more difficult problem to solve and focus their answer on solving the underlying problem. These are the systems thinkers – they know to look at the whole rather than the parts and know that understanding interactions and patterns are keys to good problem solving. The great testers – and there are only a few of these – can actually solve the problem reasonably well (frankly, I worry about testers recognizing the problem more than solving it, but I’m frequently impressed by testers who nail every aspect of this question).

nitpickers moment – for those of you who will take this opportunity to gripe about SDETs, no part of solving this question relies on programming skills. It does require that you can think and see beyond the obvious. I’m not going to put the question on my blog, because I still want to use it. I would be happy to discuss it with you privately (or via an IM session) if you’re insanely curious.

There are numerous other skills I look for in testers, but I consider those to be supportive and of the “more is better” category. For instance, if you are completely disorganized, you may not be successful, but you don’t have to be the most anal note taker either. You need some degree of organization, self motivation, confidence, and trust to be successful, but those really only show up on my radar if you truly suck at them.

As I re-read this post, I realize that the things I mentioned above are also the things that make testers successful in the long term – so it makes sense that’s what I look for when hiring testers. And I think that’s good!

The days that I’ve been at the office have been packed with human interaction – group meetings, one on one meetings, interviews, and hallway conversations.Meyers-Briggs tells me that I’m an introvert (and they’re absolutely right), but after being away from my coworkers so much lately, it was something I really craved. I also noticed that I generated a lot of work items out of these meetings. However, people didn’t give me work to do or ask me to do anything, but through the interaction, I discovered work that I needed to do. I realize that this is unique to my role on the team and that some remote workers just want to be left alone, but for me, it was energizing.

I should note that my job is a lot about discovery. I am trusted to figure out what I should be doing – or prioritize from a list of things I should or could be doing – then do those things. I reprioritize every day (sometimes multiple times during a day, and have a pretty good success rate at picking the right things to work on. It’s a great role, but it’s definitely not for everyone.

So, here I am away from the office again, but my todo list is absolutely packed. And that’s as good of a reason as any to write a short blog post today.

But not until I share this comic from The Oatmeal on the subject (because it’s funny and relevant).

First off, I can say that I was highly productive – I got a huge amount of work done. But – I have to admit, it wasn’t the same work I would have done if I was in the office. I worked on a lot of semi-deep technology problems (e.g. tools, implementations, strategies, processes, etc.) that I would have done in spurts over the next several months. It’s nice to have the research and groundwork done now, and in the long run, it will be beneficial that I got a jump start on this work. I just would have done less of it and more interacting if I was in the office.

I realized this after just a few days of work, and have spent most of the time since then pondering why this is the case. A big part of this is my role on the team. I don’t own a testing area (or even a testing technique). In some ways, I’m a consultant for our test team – answering questions, giving advice, and providing guidance where it’s requested or needed. But the questions and advice and guidance rarely come out of a well formed email. Most develop out of informal conversations – many of which I stumble upon in the hallway, by the coffee machine, or in the lunch room. When you’re working remotely, you usually don’t have good opportunities to take part in these conversations (note, that I have ideas now, and we’re working on them internally).

I attended meetings remotely (video and desktop sharing worked great), and for the first time in my career, I actually wished for more meetings, as I was able to participate just as if I was in the room (including my typical smart-ass comments). But by the time I finished the two weeks of remote work, I was starting to feel pretty isolated. I had proposed a few discussion topics to team members, but surprisingly, most preferred to wait for the discussion until I returned (I think that’s a redmond culture issue that we need to think about). I did spend almost the entire first two days I was back meeting with people and talking about all of the sorts of things that don’t end up in email, so it was nice to feel immediately connected.

My hunch is that working remotely would be a no-brainer for people with more “task-oriented” work (e.g. a job solely focused on testing, development, or writing) – as long, of course, as they had the discipline to focus on work. I’ll have to track down a few industry colleagues who work remotely some time and ask about their experiences and thoughts.

I also think that remote work can be an option for me – but it will take some culture change (and technology tweaks) to be completely successful. As it turns out, after those two days in Redmond, my parents needed me to visit and help out for a while, so I’m working remotely again. Even two days in, it’s been a better experience, and think I’ll continue to learn a lot (it also helps that I’m just two hours away, and can, and will, commute to Redmond frequently when needed).

If you have your own thoughts or experiences, I’d love to hear about them.

I’ve been working on my ET skills for over 18 years, so it’s nice to see so much recent emphasis on the approach in blogs and books. As I said, I think it’s the core of good testing, so a strong foundation in ET can only help testing improve overall. As with any hot topic in any field, there are many strong advocates, and there are definitely “camps” of thought on what exactly ET is. On a side note, those of you who know me, know that I am certainly not afraid to take jabs on just about any of the subjects I’m passionate about – both in person, and in this blog. I once made a small (a few words) comment in a blog post about ET and “belonging to a club” that brought down a deluge of email reactions that I still ponder frequently. To this day, I both regret the remark (it was a thoughtless jab), and remain somewhat dumbfounded by the reactions to the comment.

Anyway…I recently introduced ET to my (still sort of new) team at Microsoft. Actually, I didn’t really introduce it as much as I revealed it, as I discovered a natural talent throughout the team that went far beyond what I was able of teaching them.

But let me back up a bit…

I have given several presentations to my team on a variety of testing topics. I’m a big believer in testers having a big “toolbox” and knowledge on when and where to use those tools. I noticed early on in my time on the team that some testers would get so caught up in “running tests”, that they forgot to engage their brains and think about what they were doing. I also noticed that although most testers were experts in their feature area, that some had limited knowledge of the rest of the product. So, at a regularly scheduled brown bag (lunch time tech talk), I gave an intro / overview on ET. To follow up, I asked if there were any volunteers who would like to take part in an ET session with me to practice (with the secondary goal of learning more about the overall product). I quickly had a group of four volunteers, so I set up a time, and we were off to the races.

The format was:

• 90 minute meeting.

• I took the first 10 to talk about our goals for the meeting (Learn ET, Learn about the product, and Learn about tools that may help us with ET). Finding bugs is a probable, but not necessary side effect of these goals.
• For the next 75 minutes, we tested together.
• Last 5 minutes was a quick debrief and sharing of thoughts.

At the first session, I didn’t know at all what to expect. I didn’t know if people would learn, and didn’t now if we’d find bugs (I worried that if we didn’t find bugs that people wouldn’t see it as successful). I was also worried about engagement – what if people got stuck and “checked out”?

It turns out that I didn’t really need to worry. The room rocked with engagement. We agreed on random place within the application to start, I threw out a few ideas, thought out loud for a moment, and before I knew it, I couldn’t keep up with the comments and bugs and excitement in the room. I was blown away how well it went (again, it was nothing to do with me – I just pointed them in the right direction).

Based on the success, I tried another session. The same worries came to mind. The first session went so well that I had a high bar to live up to, and figured that it may have been due to the “early adopters” who signed up so quickly for the first session. Once again, I was proven wrong as a completely different set of people filled the room with ET energy.

Since then, I’ve moderated four more sessions (including one via teleconference ) and have had similar results in each of them. Better yet, some of those attendees have conducted their own ET sessions within their own team (and had similar success). Individuals are also using the approach outside of specific sessions.

Some points to share include:

• We’ve had 3-4 attendees (plus me) for each session, and I’m pretty happy with the learning experience. I think this size group is big enough that people can learn from each other rapidly, and small enough that everyone gets to be heard. I also don’t think I could keep up on notes with a larger group.
• The session length also seems to work well. People are engaged throughout and there’s barely a slow down before the session ends (one attendee mentioned that they felt guilty because the session didn’t feel like work!).
• Everyone should get used to thinking out lout – especially early in the session. It helps people learn and build off of each other’s ideas
• The focus on learning has worked well for us. I think that anytime testing focuses on finding bugs, that it veers off track

And that’s pretty much it. We’ll definitely continue to hone our ET skills, and I’m sure we’ll continue to experiment, add to our skill set, and tweak our ideas and processes as we go.

But how do I answer the question for “what’s the coverage of your testing?” for a multiple component-based application, consisting of a C/C++ major component, an application server, and customized business logic?

The answer is sort of in this post, but it’s worth elaboration.

First and foremost, if someone asks “what’s the coverage of your testing”, you can answer in multiple ways. You can say “we’ve tested the requirements”, or “we have an average of 70% line coverage throughout the product”, or “we’ve covered all of the key customer scenarios”. But what you probablyshould ask is, “What do you really want to know?” Explain that “testing coverage” probably isn’t the best way to think about it, and “identifying missing tests” is a better train of thought.

The sample I used was a simple command line app, but we usually test much larger systems, so let me also explain how you could scale code coverage from my previous example to a larger application. The solution I prefer isn’t complex – you just start at a high level, then drill down until you get to something actionable. Let’s say my overall product code coverage is 60%. That tells me that 40% of my code is untested. For the sake of easy math, let’s say the application has a million lines of code. That’s 400,000 untested lines of code. Which ones do we look at first?

Let’s drill down into the four main feature areas of the (fictitious) app.

OK – I notice that the Utilities only have 50% code coverage, but it’s the lowest priority, so I won’t start there. Instead, I’m going to look for testing holes in our pri 1 area (Engine). Let’s dig in again.

Protocols are the worst (although we’ll definitely need to look at Filters as well at some point). Digging in further tells us …

Looking like I’m missing test cases for Models…

oh wow – I’ve barely tested the functionality in one of the files (File1.cpp). One step deeper confirms the story.

From here, I’d look at each of the functions in this file and see what testing I’m missing that may help me hit this code. Once again, all I’ve done is used the code coverage data to guide my testing. Once I’ve looked at what I may be missing in File1.cpp, I may back out and look at Format Codes, or I may back all the way back to looking at Filters. I still use my knowledge of risk and priority to tell me where to investigate, then use the coverage data as a tool to help guide me in the right direction.

An important thing to reiterate, is that I’m still not concerned with improving the code coverage number – I just want to use the data to see what I’m missing. Sure, the number will go up anyway, but it’s not the point. The point is reducing risk by identifying missing test areas.

Would love to hear how you check code coverage of manual testing, actually. Is that just “feature coverage” (break it up into logical blocks and test those)?

I realized that a lot of people may not be measuring code coverage during manual testing – but it’s quite valuable to do so, as measuring code coverage still helps you find missing tests.

Let’s consider a silly command line app that does math operations. It takes 3 parameters, the first two are the values to operate on, and the third parameter is the operator to execute. For example, a command line of 2 3 + would return a value of 5. Let’s test it (manually).

c:\example>calul8r.exe
Usage:
calcul8r [value] [value] [operation]
for example:
  calcul8r 4 2 +

c:\example>calul8r.exe 2 3 +
The result of 2 + 3 is 5

c:\example>calul8r.exe 4 2 -
The result of 4 - 2 is 2

c:\example>calul8r.exe 5 5 *
The result of 5 * 5 is 25

c:\example>calul8r.exe 15 3 /
The result of 15 / 3 is 5

OK – looks like it sort of works. But are there test cases we forgot? Let’s look at what code coverage tells us.

The “meat” of this app is in the DoMath function, and the results show that I missed a few lines of code in my first round of tests. The first two lines I missed (lines 9 and 15) are a bit of discovery. It looks like the app can take the letter x as well as an * (asterisk) for multiplication, and a backslash (\) as well as a forward slash (/) for division. The functionality is the same, but it’s interesting nonetheless.

Oh – but I was stupid and forgot to test for divide by zero. It’s nice that there is some error handling for that, but if you look closer, you’ll see that executing this particular test would reveal a typo in the output string.

I also forgot to test with an invalid operator – and wouldn’t you know it, there’s another typo there. What I’ve done is used the code coverage data to guide my design for new tests. This app is simple, but the idea works on a larger scale too. I’ve known testers to discover large areas of functionality they weren’t aware of before, or (more often) hidden functionality within the areas they had tested extensively.

An important thing to remember is that once I’ve added those test cases (and reported those bugs), is that I’ll have 100% code coverage. Remember though, that 100% code coverage doesn’t mean bug free. In fact, this app doesn’t check for integer overflow. Sure enough, check this out:

c:\example>calul8r.exe 2147483647 1 +
The result of 2147483647 + 1 is -2147483648

Oops! Remember – just because the code is “covered”, it doesn’t mean it’s tested. Please don’t forget that (and don’t let your managers forget either).

Thanks Tim, for prompting this post – I hope you (and others) find it helpful.

As a quick aside, we measure code coverage using block coverage (block coverage is similar to line coverage except that it groups continuous non-branching statements into a block for counting purposes). I’ve found that bullseye has a very good primer on code coverage measurement for those who are interested in more than I feel like writing about today.

A small group of us worked on the toolset, process and strategy for our team’s use of code coverage. One topic that came up was what target percentage of code coverage we should set as a goal. Years ago, when I was on the CE team, I helped build and deploy our code coverage toolset. Once our tools were working and we measured coverage, we set a goal for the next release (65% if I remember correctly). We cranked up the number a few percentage points each release, and I think we were approaching 80% by the time I left the team. As I type this, I remember something about making a promise to shave my head if we got higher than 80%. That’s only funny because about a year ago, I did shave my head (it’s since grown back).

That history is interesting, because I had been anticipating the question (target for code coverage percentage), and I was (and am) adamant about the code coverage goal I would like our team to shoot for.

I’m adamant that we have no goal for code coverage.

The problem of having a target goal for code coverage is that code coverage will improve – i.e. you get what you measure. Wait a minute – shouldn’t improving code coverage be a good thing? It is (for reasons I’ll explain lower), but here’s what usually happens in practice.

Let’s say I own three features – one major feature with high customer impact, one area with low customer impact, and one that’s somewhere in the middle. Now, let’s assume that my team has a goal of 70% code coverage (measured as an average across the components I own). I measure code coverage, and here are my results.

Now, if your goal is to get your average to 70% (or even if your goal is a minimum of 70%), you are going to put your testing efforts into testing medium and low impact areas. Shooting for a code coverage goal can convince testers to throw out their prior knowledge of risk and impact, and instead focus on “improving the number” – by testing stuff that probably doesn’t need more testing, while ignoring potentially important stuff. If you never measured code coverage in the first place, would you really spend time doing additional testing on the medium and low impact areas of the product (with more effort needed for the low impact area)?

It’s time to discuss the goal of measuring code coverage. It’s foolish to think that higher code coverage has anything at all to do with product quality. It only measures whether code is executed on at least one path. One of my common “Alan-isms” is this:

The only thing that 80% code coverage tells you is that 20% of your code is completely untested

What code coverage does do is point you to holes in your testing. The goal of measuring code coverage is helping you (as a tester) understand what is not being tested. Once you discover what’s not being tested, you can make a choice based on risk and impact on whether you need to add additional tests – or if your time is better spent elsewhere. By not having a percentage goal for code coverage, I hope the team can focus on improving tests and testing rather than improving a number.

By the way – you can replace the words “code coverage” with “test automation” above and tell a pretty similar story.

Another thing we’re starting to do is measuring code coverage on check-ins. We’re fairly late in the cycle now, so we want to be careful of regressions and ensure that all of the code coming into the product is well tested. What we do is filter our code coverage view to only look at changed lines of code in the changed list. Say we have a binary that has 10k blocks of code, but the latest checkin only changed (or added) 25 blocks. We can filter our testing and coverage on just those 25 blocks and ensure that we’ve at a minimum executed each line at least once during our initial testing. This helps a lot with regressions, as we can ensure that we look at error cases and other little used paths very close to the checkin rather than waiting until those errors pop up at a much later date.

This gives us the potential for 100% code coverage on every changed line of code.

But I would never make that a goal :}

The broken bullet is the backwards version of the silver bullet. In the broken bullet anti-pattern, people dismiss ideas, approaches, etc. just because they are not a silver bullet. I caught myself applying the broken bullet a few weeks ago in a discussion about GUI automation – I don’t like most GUI automation because it’s fragile and rarely achieves enough return to justify the investment and maintenance, but I made the mistake of dismissing GUI automation as a solution just because I knew it didn’t work everywhere, and it was easy to get wrong – even though it could have worked well in this particular situation. Fortunately I caught myself before I made too much of a fool of myself.

Unfortunately, many others seem to embrace this anti-pattern regularly. The conversations usually go something like this:

Tester: hey everyone, I’m checking out floober as a test approach – seems like it will help me

Broken Bullet: don’t waste your time – floober is a mostly a myth and doesn’t work unless you use your brain. Here, I wrote a paper…

Tester: thanks for the help – I’ll go back to what I was doing before

Broken Bullet: no problem  – glad to keep you on the right track

Sometimes it’s more proactive. I can’t go a week without seeing an article or blog post saying “Don’t do X” – “here are all the ways it can go poorly for you and ruin your product / team / company / life. Stay away and don’t even think about X”

The problem is, that X (and floober for that matter) do work (if used carefully), and may be good solutions for some teams (and likely great solutions for others) – but will likely never get the attention they deserve because of broken bullets.

My call to action (if you care) is this: If you believe in No Silver Bullets- that there are no magic solutions to solve your software challenges, then you should also believe that there are no (ok, few) universally bad practices. Some practices are indeed much easier to get wrong, but that should only scare you – not stop you.

And the next time you see someone dismiss something because it can fail, tell them to take their broken bullets and leave you alone.

Yesterday was my first day back, and I was able to get my inbox back to zero messages and make a pretty good dent in my to do list. I expect to get the backlog under control today and start making forward progress by the end of the day today or by tomorrow for sure.

I was able to get a chunk of reading done last week. I re-read Seth Godin’s Tribes and Pat Lencioni’s Five temptations of a CEO. I also finally finished Robert Austin’s book on measuring performance in organizations (additional comments on all of these in future blog posts). I’ve also done a lot more thinking about the state of the state of software testing and expect to discuss my thoughts openly in the coming weeks.

Some random vacation stories follow…

After a 4 hour flight to Chicago, followed by another 8+ hours to Paris, we had 24 hours to kill before taking a (very fast) train to Toulouse. My wife and I have very different approaches to dealing with jet lag. My approach is to get out in the daylight and get some exercise, and she can somehow sleep it off and adjust almost as quickly. The six-year old was up for an adventure, so off we went while the girls slept. We tried the Louvre first, but Cole just couldn’t handle the line. He did, however, suggest walking to the Eiffel Tower. I like to walk, so we went for it. Eventually, we made it, and the little guy decided we should hike the stairs. I tried to talk him out of hit, but he wouldn’t have anything to do with it. The line for the elevator was at least an hour long, but within 5-10 minutes, we were climbing stairs. I figured he’d get tired, and we could turn around. Nope – he made it to the first level without a break. After a quick pain au chocolat, he was ready to climb to the second level. He was completely ready to climb to the top, but the line was at least an hour long and he was getting tired, so we headed down. We agreed that a taxi was probably the best way to return to our hotel, and we arrived back around 5:00pm. We both took a short nap then woke up the ladies to head back out on the town. We walked over to Notre Dame, saw the Louvre again (the outside), and introduced the children to the joy of chocolate crepes.The next morning I grabbed the kids as soon as we got up and headed back to the Louvre (travel tip – there’s never a line in the morning). The kids and I had a whirlwind tour (pretty much walked straight to the Italian painters and showed them the Mona Lisa), then browsed a bit more before heading to the train station.

Last week we visited Carcassonne (kids loved it). It was a little “touristy”, but not nearly as much as I expected it to be . We had a great time yesterday at the Labyrinthe de Merville. The “labyrinth” is a huge hedge maze with a twist. Along the path, there are clues that you need to read – the clues help you solve puzzles, and you use the answers to the puzzles to unlock the big gates in your way (by entering the answer on the keypad). It was fun for me because I love puzzles, and the kids couldn’t get enough of it. We did pretty well, but got stuck a few times when my French failed me (forgot our dictionary at our house in Toulouse– oops). If you’re ever near Toulouse (or Bordeaux for that matter), I’d suggest checking this place out. We may even go back again before we head home.

There’s lot’s more, but nothing terribly interesting – if you want more thoughts, comment below or bug me on twitter.

I do think that I need a break from the blogosphere and twittterville. I’m immensely excited about the number of new test blogs and test tweeters appearing these days, but I’m sort of bothered by the lack of advancement in testing ideas – it seems that people just keep on talking about the same things over and over, and keep on rehashing old arguments or statements rather than exploring new test ideas. I get it – if you do stuff wrong, it doesn’t work. There are no silver bullets. Testing is hard. Don’t insult my intelligence by telling me those things again. Even if you use all caps or new metaphors, it’s the same statements again and again and I’m getting bored.

I’ve (almost) always taken a high road with my community participation. I sometimes see popular testers tell half-truths or exaggerate their accomplishments in their diatribes to the masses, but it doesn’t seem right for me to call them out and set the record straight (even if I frequently wish someone else would). I’m not about to make every post a rant, but may not bother to try to appeal to a wide part of the testing population either. I guess the point is that I’m not seeing what I want out of the community or my own blogging, so I’m hoping that by not reading or writing blogs or tweeting for the next few weeks that I’ll figure out what I want. If not, I’ll just take more time off until I figure it out.

Feel free to leave comments – if you haven’t commented on this blog before, comments may sit in the moderation queue for a while. The rest will go through unedited until I get around to playing with the internet again.

Edit – 3:00pm, July 19

It doesn’t matter who I’m referring to above – if you think I’m talking about you, I probably am not talking about you – unless you’re sure I’m talking about you, then you’re only probably wrong.

Also – I’d like to point out that I’m a huge fan of all testers actively engaging and learning about the craft. Groups like weekend testers and individuals who explore new approaches are high on my “love” list. Also note that many who think they are actively learning and engaging actually aren’t doing either. If this statement bothers you, please refer to the above paragraph.

So let me share what I’ve been thinking about…

Every bug is more difficult to find than the previously found bug. There will be little hops off the axis, but in general, I believe this to be true. Imagine if the opposite were true – if every bug were easier to find than the previous bug, we’d find more bugs every day and eventually have a product consisting entirely of bugs (insert joke about your least favorite product here).

As bug discovery decreases, product quality improves. There are plenty of reasons where this may not be true, but if few bugs are being discovered, it probably means that product quality is improving (or that all of the testers went on vacation) – but …

Tester skill and knowledge increases with each found bug. When a tester finds a bug, they also acquire knowledge. They learn something about the product, or a technique or behavior that they can use to find bugs in the future.

Tester skill can’t grow as fast as bug discovery difficulty decreases. If tester skill improved faster than the difficulty of finding the next bug increased, we’d find more bugs over time until we had a product with more bugs than …yes – you see where I’m going.

<some made up line charts would do well here, but I’m feeling lazy>

What I’m struggling with is coming up with a better understanding of how these statements interact. In more concrete terms, if the number of bugs discovered in a product or feature area isn’t tapering off at “an expected rate”, does that mean that product quality isn’t improving as much as expected, or that tester skill is improving more than expected? Since the answer is (you guessed it) “it depends”, how do you dig deeper? How do you know the right answer with some degree of confidence?

I have answers for this that satisfy me (customer data is a big part of this), as well correlating a bunch of other measurements, but I wonder if anyone else thinks about stuff like this and has other thoughts or ideas or experiences to share.

Managers

Yesterday, I wrote about the non-management career path for testers at MS. There is, indeed, a career path for managers – I just tend to talk about the non-management career path because that’s where I’ve tried to keep myself. We have two titles for people managers at Microsoft. We call managers who manage a team of non-managers Leads, and call managers who manage leads Managers. In other words, a Test Lead manages testers, and a Test Manager manages test leads. Got it?

Generally, Test Leads are in the Senior band, although it’s somewhat common to have leads in the upper ranks of the SDET IIs. The level of knowledge (and more importantly, amount of different testing experiences) testers have by the time they get to these levels gives them the confidence, ability, and credibility to be successful. Leads are expected to get some “real” work done in addition to managing their team. Test Managers are typically Principal or higher (although, as with Test Leads, may be in the upper bounds of Senior).

Something else to note is that the Lead role does not necessarily need to grow to the Manager role. If you are a lead, but never want to manage a large team of leads, you can grow to Principal+ levels as a lead. Or, you can be a lead for a while, bounce back to being an individual contributor, and bounce back to being a lead if you desire (and if the opportunities and business needs are there).

Another take on career stages

I recently re-read Spolsky’s Smart and Gets Things Done. That (the title, and the description from the book) is pretty much what SDETs and most of the SDET2s do. Senior SDETs (and SDET2s who are on their way) move to a stage of Smart, and Makes Stuff Happen. It’s one thing to know how to do stuff really well, but leaders are able to make stuff happen through influence (and the help and support of others). We expect Principal’s (and those about to be Principal) to think more about How Stuff Happens. Strategy, decision making and and solid systems thinking all feed into the “how”. Of course, you still need to dive into making stuff happen and getting things done once in a while, but setting the right direction, for the right reasons, is critical.

Eric, a former manager of mine has a nice post on our “career stages” in this blog post. His post is on SDEs, but it applies to SDETs for the most part. Eric’s descriptions are wonderful, and well written. At risk of being a copy cat blogger, however, I’m going to offer a sentence or two of commentary on the non-manager SDET career stages at Microsoft.

SDET – This is where most of our testers start, and where you figure out how to do your job, and do it well. This stage is about learning the trade (tools, practices, dynamics, etc.). You won’t make it out of this stage just by being a fantastic tester – everyone in this stage is a fantastic tester.

SDET II – At this stage, the big thing is independence – SDET “2s” get stuff done, and don’t get stuck – in fact, they don’t let themselves get stuck. At this stage, testers also start to get their eyes of the immediate tasks at hand more often and think a bit about the future of their org. They have influence in their team, and often accomplish significant tasks that impact their entire org.

Senior SDET – Leadership is expected at this stage. It can be technical leadership or strategic influence – but the big point is that you have a huge amount of influence and impact on your entire team (with occasional forays into impact across your organization). For a longer explanation, please meet Alecha, Sanjay, Jodi and Kaz.

Principal SDET – SDETs at this level are expected to have influence and impact across their group (i.e. well beyond the testers that work in their own managers organization). We don’t have personas for this level shared externally yet, but if you think of the gang of four above with a much wider scope of impact, you’ll be close.

Partner SDET – This is scope of impact ratcheted up to 11. These folks leaders across an entire division (remember, that we’re talking about leadership – not management). They often make decisions that can save millions of dollars (and do so with consistency and confidence). Again – we have no personas for these folks, but just think of the same gang of four leading successfully across an entire division.

Internally, we have career stage profiles that define a lot more about these roles (some examples are in chapter 2 of hwtsam). The one bit of possible weirdness with the big emphasis on career paths is that we expect everyone to grow to at least Senior, and I worry sometimes that there are fantastically awesome wonderful super-testers who just aren’t capable of leadership at the Senior level. As I think about it though, I wonder if I (we?) just need a broader view of what leadership in testing really is. Right now, we have a relatively small percentage of folks at Senior+ levels, so I may be looking at this through eyes stuck on the small population of people currently “leading” at those levels.

Just hoping to clear up some mysteries – hope you find this helpful.

• I’ll be heading to Toulouse in a few weeks for a bit of a vacation. Looking forward to the time off, as well as food, wine, and warm weather. At one point in my life, I was almost adequate in speaking a very broken form of French (one that only included present tense verbs and lot’s of food terms) – but I’m pretty sure I’ve forgotten even that little bit of the language these days. Hopefully enough will come back before I embarrass myself too much.
• My “new” job (I’ve been here 4 months already) is going great. I’ve had some pretty good success with exploratory testing on the team lately, and it’s an effort that a lot of people are seeing value with. I’m also working on some stuff with code coverage, test code quality and code reviews that are keeping me busy.
• Buried in this post from just over 4 months ago was a comment about my rising blood pressure. I’m happy to say that as of this morning (after a visit to my doc), my bp has returned to it’s historic norms (actually, a bit better – 110/66). Better yet, I mentally and physically feel better these days (and this is before my vacation).

I have a follow up on my SDET Pendulum post drafted, and will try to get it up by the end of the weekend (for those of you who like my longer posts).

Too often, people see “politics” as the evil underbelly of the corporate world, and that competency in office politics requires that you can backstab your coworkers with no remorse, find a variety of ways to undermine your colleagues, and take credit for the work of anyone who won’t fight your claims. This definition from dictionary.com covers it well:

to deal with people in an opportunistic, manipulative, or devious way, as for job advancement.

But to me, that’s a cynical, shallow, and completely unhelpful view of what politics really are in the workplace. However, I do like the first four words – “To Deal With People” – in my happy rose colored view, that’s the part of the politic game that everyone has to play. I think that long term success in any workplace requires that you can deal with people. If you want influence, you’re going to need to have allies. If you attempt a difficult change, you may also annoy or disappoint some people. That’s all fine, as long as you find a way to talk, listen, or do whatever it takes to keep the wheels moving. I’m not saying that you have to kiss everyone’s ass, but if you want to be a leader – someone who can influence people to change, improve, or flat out take a chance on something that you want to do, you’re going to need to find a way to deal with them.

There may be people in the world that you don’t like or respect. That’s ok –we all know those people. Those of you who are politically savvy know that you probably still need to be able to communicate with them occasionally, so you don’t burn bridges. Who knows – you may need those people as an ally someday, and if you flipped the bit on them (and told them that you flipped the bit as well), all I can say is good luck getting that next “big idea” to fly.

More importantly, you need to be able to influence the people who can help you. Chances are good that you have more great ideas than you can take care of yourself. That’s ok, because great leaders take pride in making those around them better, and your great ideas are probably the thing that will make all of your colleagues better.

As long as they’ll listen to you.

And they will listen to you, as long as you are worth listening to. So build credibility, help them in their own projects, give them credit, and raise them up any way you can. In other words, use politics (the good kind) to get your way…and become a leader.

That’s pretty much it, so I suppose I can end this post here…

But one question came up that I want to dig into a bit, and I know that some of you who occasionally read this blog have some opinions on the subject, so fire them away please.

The question in mind may as well have been seeded by me, because it’s a subject I’ve brought up before. The question was: “Has Microsoft swung the pendulum too far in hiring only programmers as testers?” I can’t remember if my answer was “No, but Yes”, or “Yes, but No”, or “It depends”, but I’ll give a (much) longer version here for your benefit.

BTW – I would bet that if I asked most non-microsoftees the response they would give for this question, they’d say, “of course – my {cousin | dad | sister | friend } who is the best tester in the {company | city | country | world } can’t get a job there, and they’re {fantastic | awesome | superman }. They just can’t write code.”

And, in many cases, they may be right – but maybe not – or it depends. Some context is in order. I’m not the company expert on this, but my opinion and view of history is as good as any.

Our software is big and complex. We deliver platforms, operating systems, server products and put them into huge deployments. We need testers who can program to help solve the big testing problems that our systems present. Say, for example, you’re a tester on the Exchange team, and you need to verify that messages of varying lengths travel from one endpoint to another successfully. No problem – send mail to a test account and make sure it gets there… well…that solves one happy path. The reality is that the tester needs to send multiple mails through the system with a variety of headers and sizes to ensure that anytime the message is broken up that it still ends up at point B correctly. And, going through outlook is probably not the most efficient way to test this, so it’s probably better to construct the messages programmatically then send them through the system. Oh yeah – there are an almost infinite number of ways the server topology can be constructed, and those settings make a difference too (in other words, we need to deploy any arbitrary topology automatically on demand for these tests). As you can see, there are some obvious benefits to having someone who can program as a tester.

Microsoft has always had testers that can write code. Always. For much of our history, we’ve also employed testers who don’t write code (but who are still good testers). We’ve found over the years that our best testers and best test leaders had a programming or CS background. I’m guessing we also found that defining the career path (MS is really, really big on career paths) for testers was a bit easer to map out when we took programming background and application into account. We still use a lot of non-programmer testers in contract and vendor roles, and I’m guessing we’ll continue to do so. There are many great testers who have learned how to program, but over time we found (or I should say, my personal experience was) that when we fished in the pool of programmers for good testers, we found a much higher percentage of good people than when we fished in the pool of good testers looking for people who could write programs. So, for full-time “blue-badge” employees, we fish for testers where programmers live; we get what we want, and everyone is happy.

Hardly.

Note that in the above I referred to testers who can program. Where I fear we make the mistake in hiring is when we (or anyone else who hires “SDETs”) hire programmers to do testing. The difference is subtle, but important. A tester who can program is a tester first – but they are a tester who can rely on computer knowledge or programming skills to do their work better. A programmer who tests writes tools and automation first – hoping that it will help them be a better tester (hint – it doesn’t work very often). For the most part, Microsoft has testers who can program – and they are freakin’ awesome. The problems they solve and the way they go about it makes me proud, excited, happy, (and a bit jealous) all at once. It’s what SDETs should do.

The industry perception of the SDET appears to be a role of writing automation and tools all day. When I hear this, I think of all the great testers I work with and tell people “No, No – just because our testers can program doesn’t mean that’s all they do all day – they are testers first”. Then, whether it’s in 5 minutes or 5 days (but sometime soon after), I’ll see a blog or a tweet or some other sort of message from someone saying “I’m an SDET – I write automation and tools all day”, and I realize that there’s still some work to do. It doesn’t mean that most of our SDETs write tools and automation all day, but there are certainly pockets. My thoughts are that if you want to write tools and code all day, get out of my business – go be a developer. You’re not helping me (and you’re probably slowing me down), so leave me alone. I want to test, and I want to work along side people who feel the same way.

In the panel discussion I asked every hiring manager in the room to ensure that in the next interview they conducted that they looked for testers who used “technical” skills to solve their problems. Instead of making them write code on the whiteboard, give them hypothetical testing tasks and examine how and when they used the power of the computer to help them. For example, if I ask an interview candidate how he would know if a line drawn on the screen was straight and he started writing up an algorithm to scan the pixels on the screen and determine if the offsets were consistent with a “straight” line, I wouldn’t hire him (“Look at it”, or “use a straight edge while looking at it” are both acceptable answers). However, if I describe the message transport scenario I used a few paragraphs above and their answer is to ask everyone in the company to send a bunch of mail, I probably won’t hire you either. (I’m working on an internal presentation on hiring testers – I’ll come up with better examples :})

Every business is different, and your need for testers or “SDETs” will depend on your business. My call to action is this: If you wan to hire an SDET – hire a tester who can program – not the other way around”. You and your product will be much better off.

I’ve never met Joe – but I feel like I know him a little. I interact with testers regularly that I’ve never met. I take time every day to see what testers are doing, what they’re trying, and what they’re thinking. The problem is, that if you’re reading this, you’re probably the same as me. You look beyond the walls of your job to discover new ideas about testing. The talk I helped with (Joshua Williams actually ran the talk) was about looking outside the walls of Microsoft (or wherever you work) to discover what other testers were doing and what they cared about. The best testers I know have a passion for learning – but more importantly, they seek out new ways of learning, rather than just reading one source, or following the work of one test blogger or consultant.

So – I gave them options. I talked about the testing stack exchange, software testing club, test republic, and sqaforums. I talked about Software Test & Performance…I mean Software Test & Quality Assurance magazine and Better Software. We have a license for the IEEE Explore site, and I unveiled the world of academic papers on testing (and above all stressed the importance of critical thinking when attempting to suck down this info). I tell a story sometimes about the first testing book I read. It opened my eyes – after reading it, I felt like I knew everything (I was too naive to challenge it). I read a second book on testing and noticed that it contradicted the first book sometimes. It wasn’t until I read my third (and fourth, fifth, sixth and more) that I began to form my own opinions about testing. Forming your own opinions – rather than just repeating what others say, is something I value tremendously in testers I work with. I also gave a nice plug to weekend testing, as the movement is all about learning and testing.

We also talked about open source. We’re not big adopters of oss at Microsoft, so a lot of people don’t know much about what’s out there. I said what I could without pissing off our lawyers too much.

That was pretty much it. There were some good tangents in the discussion section. Somebody was disappointed that we just didn’t tell them what was happening in the industry (you’re missing the point), but I hope we at least opened a few people’s eyes to the world of testing.

If you’re reading this, this post probably isn’t for you – but get your coworkers (just the one’s you care about should be fine) to look beyond their little part of the world to learn. It’s a big world of testing out there.

With my remaining hours, I do “other stuff about testing” (I’ve been thinking about calling this “Project OSAT”, but it’s probably better to just call it “other stuff”). This includes writing books, contributing to other books, writing articles and blog posts, interacting on twitter, and participating in the overall testing community as much as I can find time for.

Sometimes, colleagues ask why I write, or why I speak at conferences. It’s not fame, and it’s certainly not for the money. It’s for ONE reason, and probably because of my lack of sleep due to 4:30am world cup matches, I’m going to share my motivation with everyone.

I write and speak because I’m lazy and cheap.

I suppose I should explain…

How We Test Software At Microsoft was written for one main reason. I talk to a LOT of people and companies about testing – many want to know Microsoft’s general approach to testing. I gave many of these companies the same answers. Over, and over and over again. Finally, I decided (with the help of a few colleagues) to write it all down and sell it for 25 bucks a pop (street price). There was just no way I could keep my sanity and continue to tell people about what testers do at Microsoft. Most of the articles and blog posts I’ve written have been for the same reason – I’m too lazy to give the same answer to multiple people, so when I start hearing the same question too many times, I write it down somewhere (NOTE: this is often also a good heuristic for automation).

Let’s not forget cheap. I hate paying to attend conferences (even if it’s technically my employer’s money). I do like to meet with other testers and talk shop (aka Project OSAT), but if I can get into the conference for free, I’m all over it (sometimes I even manage to get flights or more covered). So I submit proposals to a few conferences a year, and I typically get asked to speak at a few more. I do try to do a good job when I’m there (despite my laziness, I have a high bar for personal quality), but I go to conferences mostly because I get in for free.

Just because I’m lazy and cheap doesn’t mean I don’t care. I love writing, and I love speaking about testing and sharing what I’ve learned. I have worked, and continue to work extremely hard at both of these endeavors (see the note about my high bar for personal quality above). In fact, I’ve never taken money for any writing or speaking (some of probably think this is dumb, but it’s true). Way back when I first started writing for Better Software, I did this because not getting paid was easier than trying to decipher the company moonlighting policy. These days, I’ve found that I actually can get paid for most of this stuff, but I still don’t bother with it. When I consider how much I enjoy writing and speaking, doing it for free seems like the right thing to do (as long as you give me something free too – I’m still cheap).