In the absence of information, people will make stuff up.

This is a phrase I share often with mentees and colleagues – most often when talking about progress on a project they’re working on. I once saw a colleague take a lot of heat for a project he was leading – he was making great progress and doing (almost) everything right, but he wasn’t sharing his progress – and when he did, it was incomplete. He really was doing great work – I remain impressed today at how much he and his team were accomplishing and the obstacles they overcame, but if you weren’t part of his team or closely involved, you didn’t know what was going on.

And – as I said above, without the information, people made stuff up. The situation was a bit of a mess for a while, but he learned his lesson, and his work ended up back on course (although it was never far off course anyway). The point is that if you’re leading any sort of project, you have to let people know what’s going on – that’s one reason short iterations, demos, and retrospectives on agile teams are good – it’s a method of sharing progress, and it removes a lot of the ambiguity from the process.

This sort of thing happens all the time. When a professional athlete doesn’t dress for a game, or doesn’t play, the team usually makes an announcement explaining the situation. Without the explanation, people would make stuff up (although in the case of sports rumors, people will often make stuff up even if they have information)

This morning, I was thinking about some of the thoughts and rumors about testing at Microsoft – particularly some of the musings about why MS only hires testers with coding knowledge for full time positions. There are many reasons for this move, and most of them are good reasons – but the role shift hasn’t been without its problems.

I have to admit that I somehow feel the need to offer correction when I see people post their random theories on why MS made the decision to move to programmer-testers and the role of testers at MS. It’s somewhat stupid of me to care. I can’t speak for ten thousand testers and what they do. I’m not a spokesperson for Microsoft. I wasn’t even in the room when the decision occurred. Still, I wish someone with authority would have done a better job explaining the role of the SDET years ago, because without any information on the why and how of the move, people make stuff up. And who can blame them for that?

When I decided to write HWTSAM, it was mostly to explain the role of testing at MS – at that time, I was speaking to quite a few Microsoft customers who had questions about how we tested at scale, or how we tackled certain testing challenges. After answering many of the same questions several times, sharing some of what we do in a book seemed like a logical choice. The book, however, covers a lot of the ideas we use, but doesn’t fully capture enough of the innovative ideas our testers come up with on a near-daily basis.

A prominent tester once described HWTSAM as a book about writing automation. Obviously, this person hadn’t read the book (or perhaps has a different definition of automation than I do), but I am pretty sure they never read a page of the book. In the absence of the information they would have had from reading the book, they made stuff up.

I’d love to tell a better story – one that eliminates the conspiracy theories forever…but even I probably don’t have all the answers. I can share what I know – and it’s probably more than most people, but I’m sure I’m not the right person to explain the details. When I think about it in context, it doesn’t really even matter. I want to be part of improving and advancing software testing and the role of the tester. There are plenty of valid reasons to pick on the programmer-tester role, but few of those advance our profession, so there’s probably a better way I can use my time.

Whew – what a week it’s been. There’s only drawback I can think of from being in Switzerland last week is that my work and meeting load this week has been nearly double. Given that I’m still catching up from my India trip last month, I’m quite thankful that my travel schedule is light in the coming months. I’m looking forward to taking on some fun and challenging work (which I’ll share here when appropriate).

A few weeks ago, I talked about regression tests and gave a few examples of how a tester could write automated regression tests that had more of a chance of finding new issues than typical regression tests (for the full story, read here).

In that post (you did read it, didn’t you), I mentioned the "I wonder…" principle with the example of "I wonder if this will work if I pull out the network cable?" An area where the often misconstrued "programmer-tester" can put coding skillz to work is to automate the "I wonder" part of the test. If removing a network cable during a particular transaction is interesting, then removing it during almost any network transaction would be interesting. So why not add functionality to the test automation system that throttles or interrupts network activity on all transactions – (or make it a configurable variation so you have some test runs that work with consistent network connections).

To be clear, I’m not suggesting replacing exploratory testing with automation. What I’m saying is that many things we do when exploring can be scaled using automation.

Fault injection is a pretty typical way for coder-testers to add value to automation (frankly, I don’t consider people who just write point A to point B scripted automation to be testers, but that’s something I suppose I’ll tackle in the comments section). Testing for application behavior with low disk space, low system memory, and any other common failure scenarios is a great way to make your stagnant automated tests find new issues.

In a comment, Markus mentioned fuzzing – while not necessarily a way to make generic automation find more issues, it is another place where writing automation is the only (practical) solution. Fuzzing, in a nutshell, is the act of intelligently munging any sort of data used by the system under test (note – for a full definition of fuzzing, see the wikipedia article). File fuzzing, for example, is the process of taking a valid file, and then mucking with a bit of the internal structure in order to expose problems (including potential security problems) in the way the application handles that file. (Note that what I call "dumb-fuzzing, you can take any old file, mess with it randomly, and see what happens – it’s a reasonable test, but typically won’t find as many security-class issues). A binary file type (e.g. a word doc) typically includes information about the file, including the number of sections in the file and the length of those sections). If you’re careful about how you tweak some of the bits in the headers and sections, you can potentially trick the application into crashing – or, if you’re really good, into running arbitrary bits of code. You can do the same thing with any sort of binary data – for example, we do a lot of protocol fuzzing – mucking with bits to make sure the receiving end of the protocol does the right thing.

In order to cover all of the permutations, a good fuzzing approach may require thousands of files. Editing ten thousand files in a hex editor isn’t my idea of excitement, so automated fuzzing is a good solution. Oh – and loading all those files into an application doesn’t seem that exciting to me either, so that part should probably be automated too.

There’s more stuff for the tester-coder to do. In a client server org like I work in, we have to test against multiple network topologies. Luckily, with a bit of programming, we can do that all automatically and save a bunch of time. Now – to be fair, there is some tax to having all of this infrastructure. With automated tests and fault injection and topologies and all of the other parts of the system moving, a reasonable amount of time goes to maintenance, and that part sucks, but for the most part, a team of testers with prog skills /have the potential/ to do a reasonably good job testing fairly complex software.

However – if your team only writes regression-ish automation – or if your view of the tester-developer is someone who only writes automated regression tests, you are completely missing the point of coding skills for testers. I know (and fear) that this is the case for way-too-many people in the profession today, and I find it a bit disheartening. I do hope, however, that the next generation of testers will have the big-picture thinking we need to find the harmony between brain and skills that testing needs to grow.

I spent the last few days in Zurich – it was a brief, but nice stay. I left Seattle on Sunday, and arrived in Zurich on Monday – I spent the afternoon touring the city in an attempt to walk off jet lag. On Tuesday, I gave a presentation to a group of customers and had lunch with several of the attendees. During the day, I also met a few of the fine folks from Swiss Q consulting. They’re the one’s who put together Swiss Testing Day (as well as some of the other testing days in Europe). Everyone I talked to from Swiss Q was great – they all seem to be incredibly nice people. I was pretty tired by Tuesday afternoon and was half dreaming of taking a nap or getting to bed early when Adrian Zwingli (the CEO) invited me to a summit meeting he was holding in the afternoon. The attendees were all apparently fairly senior folks in their various companies and I was impressed at the turnout. Not one to avoid talking about testing, I had to tag along. Adrian arranged for one of the breakout groups to discuss our topic in English, and our group had an excellent conversation and shared some good ideas. There was a keynote in German, but the slides were in English, so I was able to follow along. After some closing remarks, we had a few drinks and some great food (and more conversations about testing).

Somehow, I managed to get a second wind and talked some of the folks into taking me some place to watch Champions League matches. Both of the matches were close and I was a little shocked to see Bayern Munich lose to Inter Milan in the 88th minute. I’m looking forward to the next round.

I made it back to the hotel before it was too late and got some rest. Wednesday was Swiss Testing Day. I was blown away by how many people attend the event. They sold out the event at 750. That’s a huge number compared to many US testing conferences. The amazing stat is about 90% of the testers are Swiss, and there are only about 4000 testers in Switzerland. It’s fantastic that so much of potential audience actually attends the conference.

I spent most of the day in the Microsoft booth, talking with testers and signing copies of HWTSAM. I took a brief break from booth duty to give a quick test talk. A bunch of people gave me good feedback on the talk, but from my end, I didn’t do as well as I hoped (or as well as I usually do). I had a huge crowd, and that was nice, and I hope those that attended found some useful information. The next time I come back to Swiss Testing Day, I promise to deliver an A+ performance.

I hung out with the conference staff (and Julian Harty) most of the rest of the night (I left the bar for a few hours to go watch Champions League again). I found some common sense and got back to my hotel before it got too late so I could squeeze in a few hours of sleep before my 7:00am flight back home.

Overall, I’m a huge fan of the conference. The camaraderie and passion reminds me of a much larger PNSQC. There are a lot of testers facing similar problems, and everyone I talked to seemed to want to learn more about testing; and most importantly, there was a good feeling of community among the attendees. I see a bright future of testing in Switzerland, and I was honored to be able to be a part of this year’s session.

I love to travel, but I’m sort of glad that my travel schedule is almost empty (and devoid of international travel) for the next several months. Hopefully, however, Swiss Testing Day 2012 will pop up on my agenda next year.

In a recent talk on test design (link), I discussed the concept of "useful tests". In my definition, useful tests are tests that provide new information. Almost every test is useful…once – typically the first time it’s run where it shows that the underlying functionality is working. From that point on, many tests function primarily as regression tests. We run these tests as a safeguard to ensure that we don’t break anything with future changes. They’re great for this purpose – and when they find a bug, they are wonderfully useful. But 99% of the time (according to my definition) they are not useful – they don’t provide new information (they do provide information – the information that nothing has changed, but it’s not new information).

To be fair, these are exactly the types of tests I want for unit tests (and probably acceptance tests too, depending on the definition). I love using unit tests to give everyone on the team confidence that refactoring or feature additions do not break basic functionality – but unit tests (and acceptance tests) are usually just a small portion of the overall testing effort.

I should stop here to clarify. On some teams who "do automation", the unit and acceptance test automation is all they do. In this case, the regression only focus may be appropriate.

Fortunately, the teams I work on (and with) use automation for far more than regression testing – , we need tests that can be "useful" (provide new information) way more frequently than when something that was working once stops working. Test automation isn’t just for regression testing It cannot (and shouldn’t try to) replace human brain-engaged testing, but it can certainly do more than automate a bunch of rote tasks. In my opinion, an automation strategy that only performs regression testing is short-sighted and incomplete.

So – how do you write an automated test that provides new information more frequently? A big advantage of human testing is what I call the "I wonder…" principle. For example, "I wonder what will happen if I cancel this operation in the middle?" or "I wonder if this will work if I pull out the network cable?". Computer programs aren’t very good at wondering, but they are good at brute force. Model-based testing is a potential solution here, as you can use it to traverse decision points in the application (randomly, or by applying a graph traversal technique). Traversing an application in unexpected ways often results in finding new issues. Same test, different bugs – and that seems pretty useful to me. Some testers, for whatever reason, seem to be a bit afraid or skeptical of MBT. I think in many cases, potential adopters of MBT either try to do too much with it before they understand the concepts, or think that it’s a replacement for some other types of testing. MBT is just another test design technique – and like other test design techniques, it works great in particular situations, and not so well in others.

But MBT isn’t the only way to get tests to provide different information on subsequent runs. Simply introducing some randomness to a test can often turn up new issues. For example if you have a "suite" of tests that always run in the same order, try running them in a random order on each run (note – log the random seed so you can reproduce any discovered issues later). Or, say you have a test that does a particular operation five times. Instead of running the operation five times on every test, what about running the operation [between 1 & 10 times] randomly. Regression tests, for all the value they bring, tend to train the system to pass. By mixing things up, we often find new issues.

I’m also a fan of data driven testing. Get the test data out of the automation code and into a data file (use XML or another DSL as appropriate). Then, randomize the data used by the test (and automate the randomization too). Then when you’re testing a web page where users enter address information, you can easily mix up a variety of name, address, zip code, etc. data (especially including invalid data), and see what happens. If you send the same data to the system every time, it will certainly pass all of your tests eventually, but I’m almost certain that you’ll still miss issues too.

Depending on what you’re testing, I’m sure I can think of other ideas that may make your automated tests useful more often. There’s nothing wrong with regression testing, but if that comprises all of your test automation effort, you’re probably leaving some cards (and bugs) on the table.

I meant to post this a few days ago, but I just passed my one-year anniversary on the Lync (formerly Office Communicator) team. I enjoyed my time in the engineering excellence team at Microsoft, but I didn’t realize how much I missed being part of a software team until, well – a year or so ago. I’m having a great time and can’t believe what great people and technologies I get to work with.

If you don’t know what Lync is, it’s a communications and collaboration platform (IM, Voip, desktop / application sharing, etc.). Here are some videos you can check out if you want to more info (warning – they may be a *little* cheesy)

We’re in the early stages of our next release now – it’s the first time I’ve been part of this process for over eight years (that’s half of my career at Microsoft). Although many things are the same, it’s also fun to have a different set of ideas to try and experiment with as we get ready to build the next generation of a fantastic unified communications platform.

I do a lot of mentoring and coaching within Microsoft. I like to help people, it’s fun, and it’s nice that I have a manager that sees the value in that. Sometimes, the coaching I do is on improving testing ability, increasing testing breadth or brainstorming testing ideas. Most often, however, I help people think about their career in test.

The career path concept is huge at Microsoft. I’ve been a tester (in some form or another) for almost eighteen years now, but there are plenty of testers who have been testing longer (and plenty of others who are at a higher career "ladder level" than me). We have a huge amount of resources on growing careers (including personas), and we expect people to "grow" in experience, ability, and scope on a fairly consistent pace throughout their career.

The problem many testers face is that they’re too busy with their "day job" to look up and think about the future. This is as true for looking at the big picture of testing as it is for thinking about their career. I call this "The Tester Treadmill". What I do often in my mentoring sessions is try to help people look up and see a little ways into their future. One exercise I use is to ask them this:

"A year (or two) from now, when people talk about you, what do they say?"

This (to me) is a huge twist on the typical "What do you want to be doing in a year?" type question. This gets people to think about the type of person they want to be and gets them to think about how they influence others (something we value highly at Microsoft). The answer to this question also gives me a lot of insight into where people really want to go that I may not get otherwise (at least not as quickly). I do this exercise with myself every few years, and it’s surprising how little it changes. The type of work I do changes a lot, but how I want others to perceive my contributions doesn’t change much at all. For others, direction changes more often, and that’s ok too.

Let’s say the answer someone gives to this question is, "I want to be thought of as a strong customer advocate, a great problem solver, and an influential tester". My next step is to ask, What are you doing today to be a customer advocate | great problem solver | influential tester?. Then we can focus on the types of work opportunities they can look for, discussions they can have with their manager, books they can read, etc. It’s an approach that has worked well for me, and helps me focus on career and personal growth simultaneously.

Somewhere later down the line, I bring up values. Values (to me) are what drives how you work, how you interact, and how you set about accomplishing your goals. However – if I were to ask you what your values were, would you know? A useful crutch I use to help people (and myself) determine values is to think about what really pisses you off – your strongest values are usually the opposite.

I value fairness and honesty pretty strongly – I know this, because the opposites really, really grate on me.

For example, I love soccer. I play and watch whenever I can. I’ll watch MLS, Premiere League, Bundesliga, or whatever I can find. I can watch a game between two teams I don’t care about at all or know anything about and have a great time. But the moment a referee makes a bad call (or non-call), fairness is compromised and I get annoyed angry. It’s silly, but I can’t help it. I know that "life isn’t fair", but I can’t help it – fairness is a strong value with me.

Another example – you know that kid in school that everyone liked (or at least most people) – but you know that his whole reputation was based on half-truths and downright lies? The guy that made up stories about things he had done – or skewed the facts just enough that they were hard to prove wrong, but you knew the full story? That guy bugs me – I just have a really hard time dealing with people who are so concerned with being popular that they are untruthful. Truth is a strong value with me, and after a while, no matter how much I respect what you do, I can’t have respect for you if you are untruthful. We all know people like this – depending on your values, you may despise them too – or you may follow them if that isn’t a strong value for you (or you have a stronger value that puts you in a different camp).

The point of all this, I suppose, is that no matter what you want to do, I think you need a plan to get there – and you need to know what you’re going to use to get you there. Thinking about a career is tough, and probably something people don’t think about enough.

I’m in Hyderabad, India this week to meet with some of our team and to give a handful of presentations at an internal conference. The weather is wonderful this week – it’s warm during the day and cool at night. I got to the hotel around 2:00am Tuesday morning, but got a few hours of sleep before heading into work. I was pretty loopy by the end of the day yesterday, but feel pretty good today (so far).

It’s been a few years since I was last in India, and I forgot how much I enjoy Indian food. Most of my colleagues seem to be sticking with the "safe" American style food available in the hotel, but I don’t think they don’t know what they’re missing.

Speaking of colleagues, while exchanging some email with my manager (who is also here), I realized something significant about this trip. Ross (my manager) mentioned that we (there are five of us from the Lync team here) were meeting for breakfast (at some time I don’t remember), then heading over to MS afterwards. It hit me then that this is the first time since 2003 or so that I’ve taken a business trip non-solo. Due to the nature of my role in my last group along with my occasional conference appearance, I haven’t traveled with other people in a long, long time – and I bet I’ve taken at least 30-40 trips in the last 6 years alone. To be fair, it’s not difficult having others on the trip, in fact in some ways it’s easier (someone else, for example, figures out where to eat and what time we need to be someplace). It’s just sort of a new experience for me.

Each of the major MS subsidiaries (including the main campus in Redmond) has a yearly "engineering forum" – which is basically a mini-software conference. Because I happen to be in Hyderabad to meet with some of the Lync test team, I was also hooked into giving several presentations. Thinking back, the forum team here did a pretty good job taking advantage of my presence. At first they asked for one talk, so I gave them some sample topics I could talk about. In that conversation, they asked if I could give two talks. I said sure. A few days later, they asked me for abstracts for my three talks – and that’s where we stand today (if you don’t count the panel discussion I’m involved with on Friday).

My talks are all on Thursday – I’ll start off with a talk about Tester DNA – i.e. what makes some testers great. This was a talk I was going to give in Redmond a month or so ago, but ended up cancelling due to some scheduling problems. My second talk – immediately after is on customer focused test design – something I’ve been digging pretty deeply into lately. I previewed a snippet of it into a meeting yesterday and it seemed to be well received, so we’ll see what a larger scale reveals. I get an hour break after the test design talk, then I’ll talk about testing in production. I’ve been a long time fan of testing in production (TiP) ever since Ken Johnston started developing and communicating his ideas on the subject while writing chapter 14 of hwtsam. Ken, along with people like Seth Eliot, Keith Stobie, Felix Deschamps and many others at Microsoft are mega-experts in the subject, while I’m just sort of a big fan of the concept. I’m a big enough fan, however, that I’m confident I can pull off a nice 101 level talk about TiP and I’m looking forward to this talk a lot. And – since it’s the 3rd of 3 talks on Thursday, I’m looking forward to completing the talk as well.

Friday, as I mentioned, I’m taking part in a panel discussion in the afternoon. I hope to meet with more of the test team today and Friday as well – we’re also trying to organize a test team cricket match Friday, and I hope we can get that set up.

And then, just when I’m settled in, it will be time to head home. I’ll fly back to Seattle late Saturday night.

In case you missed it (because I only mentioned it on twitter), I have another article up on Sticky Minds this week.

Here’s the abstract:

Many testers have been involved in post-ship decisions about bugs that "got away"–bugs that escaped testing and found their way into customer’s hands. Often, these post-mortem discussions end up with finger pointing and threats, but with the right focus, these discussions are a wonderful opportunity for learning and growth.

 

And here’s a link to the article.

After my presentation at PNSQC in October, I was giving away copies of How We Test Software at Microsoft, and Michael Larsen said that if I gave him a copy, he’d review it on his blog. I figured that was better than giving it to someone to sell on ebay, so I took a shot.

Michael responded with the most in depth multi-part book review ever. Seriously, it was pretty cool, and I owe Michael a copy of my next book (which I plan to start writing sometime in 2027).

If you missed it and want a nice recap of the book, here’s the full review index.

Chapter 1: Software Engineering at Microsoft

Chapter 2: Software Test Engineers at Microsoft

Chapter 3: Engineering Life Cycles

Chapter 4: A Practical Approach to Test Case Design

Chapter 5: Functional Testing Techniques

Chapter 6: Structural Testing Techniques

Chapter 7: Analyzing Risk with Code Complexity

Chapter 8: Model-Based Testing

Chapter 9: Managing Bugs and Test Cases

Chapter 10: Test Automation

Chapter 11: Non-Functional Testing

Chapter 12: Other Tools

Chapter 13: Customer Feedback Systems

Chapter 14: Testing Software Plus Services

Chapter 15: Solving Tomorrow’s Problems Today

Chapter 16: Building the Future

Afterward

Thanks again to Michael for the in-depth write up. I expect that anyone wanting the short version of hwtsam will be greatly appreciative.

Michael Bolton noticed my last blog post and wrote a nice follow up outlining several factors that lead to testers getting lost in the weeds vs. finding great bugs. I had my own ideas when I wrote the post, but the breadth of suggestions from Michael, as well as from commenters on the original post were fantastic.

To me, two of those suggestions sing most true. First, Michael suggests that one problem is the overemphasis on proving functional correctness (several of his follow up points re-emphasize this). I couldn’t agree more. I worry a lot that testers are so focused on running through their short little lists of functional test techniques to “make quality software”, that they miss a lot of stuff right in front of them – or mis-prioritize what they see. Not coincidentally, this is something I’ve been thinking about (and working on) a lot lately. The discussion topic I’m bringing to the Writing About Testing conference is along the lines of "The End of Functional Testing as We Know It" – but I’m still working on a better title….as well as the essay to back up the idea (hold on Chris – it’s on my short list). My team at Microsoft is also looking at how we can lessen the emphasis on functional testing, and as we make more progress, I’ll be sure to share more.

The other bit I think has a huge contribution to this is something Joe Strazzere said in a comment

I think much of it is motivation. Some shops, and some testers, are very positively motivated toward higher bug count.

This is unfortunately too true. The emphasis on bug finding and testers who have bug quotas or who are paid or rewarded by the bug encourages testers to report anything in the world that they consider slightly out of the ordinary.

Now – this is where the judgment comes into play. Of course, if something has the potential to annoy someone, it’s a bug and we should report it..

Or should it? Look at the following (fake) dialog message.

This action has been completed. Please check your work, save any open files, then close the appIication

OK – now find all the bugs in that error message.

I can find at least four, but which is most important? Which bug do you put up on a flagpole as the one that must-be-fixed vs. the bugs you can log and then fix “later”? The answer, of course, is, "it depends", but we can talk a bit about what it depends on.

The most critical bug is probably the misspelling of the word ‘application’. If you look closely, you’ll see that the fourth letter ‘L’ is actually a capital ‘I’ (pronounced eye). Oops, that’s huge and we’ll need to fix it.

Next off is probably the missing period at the end of the sentence – but I’d look for consistency here before reporting it. In general, sentences in error messages end in a period, but I’d want to see if just this error was missing the period or if it was a preference that was more wide spread. If so, I could enter one bug to add periods to the end of all error messages. Unless, of course, I’m paid by the bug, then I would add bugs for every error message in the program (woo-hoo – I’m going to Disneyland!).

The others are more subtle. It may read a bit better to say, "…save any open files, and then close the application". How about the first sentence – doesn’t passive voice just grate on you. That must be a bug. Go ahead and insert a comment about needing someone capable of writing error messages on the team. What I worry about a bit – and what I’ve seen is testers so focused on finding any bug, that they find the passive voice and grammar bug well before they find the real bugs in this error message. That’s too bad.

If you’ve paid attention, there’s an underlying message of this post. Reading between the lines, I’m sort of saying that functional testing and finding bugs aren’t that important in testing. I’m somewhat ok with this statement, but I’m sure it will bother someone. My point is that if testers focus on making sure the software provides value to the customers, and let finding bugs (including functional errors) exist as a side effect of that focus, that good things will happen.