Without too many boring details, I had a backlog item to write a service to automatically launch a memory analysis tool and periodically take snapshots of memory allocations in specified processes. The memory analysis tools (which I also wrote adapted from another project) work really well, but need a lot of manual intervention in order to get accurate data. I wrote a quick “spec” on my whiteboard, wrote tests as I went along, and in about four hours I had it up and running successfully. I’m confident that the service will make it much easier for our team to find and isolate memory issues. To me, writing this tool was a testing activity. It doesn’t test anything, but certainly makes testing of the product much easier.

On my drive home, I was pondering the universe (and my day) and remembered that there’s a sad bit of truth in the software world. It’s a fallacy that causes great confusion. It promotes poor strategies, bad hiring decisions, and endless confusion among management, contributors, and consultants. It may very well be the most misunderstood word in the world of software. It’s the “A” word that is practically ruining testing.

Automation.

Yes – as much as I hate them, I used a one word paragraph. I did it because the abuse and misuse of “automation” in testing is causing too many smart people to do too many dumb things. That said, I suppose an explanation is in order.

Once upon a time, someone noticed that testers who knew how to code (or coders who knew how to test) were doing some wicked-cool testing. They had tools and stress tests and monitoring and analysis and they found great bugs that nobody else could find – “Test Developers” were all the rage…so we (the industry) hired more of them. Not much later (and possibly simultaneously), someone else decided that testers who could code could automate all of the tasks that the silly testers who didn’t code used to do. Instead of finding really cool bugs, coding testers were now automating a bunch of steps previously done manually. Not to be left out, UI automation frameworks began to sprout, with a promise that “anyone can automate – even a tester.” Now, test teams can spend the bulk of their time maintaining and debugging tests that never should have been automated in the first place.

And that was the first step into darkness.

This mindset, although common, kills me:

• Testers who code write automation
• Automation is primarily made up of a suite of user tasks
• Automation is primarily done through the GUI

All of these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test teams in the wrong place and is, frankly, insulting to testers who know coding.

If you’re a tester who writes code (or a coder who cares about testing), try this:

• Write tools that help you find important bugs quickly
• Write a few high level tests that ensure user actions “work”, but leave most of it for exploratory testing, user testing (assuming you have great monitoring tools), or both.
• If you must write GUI tests for the above, write fewer of them (that way you’ll spend less time investigating their flaky behavior).

If you’re a tester who writes code, you’re not a test automator, you’re not a test automation engineer, and you don’t write test automation or test code.

You write code. And you test.

You write code that helps the product get out the door sooner. And if the code you write (or the testing you do) doesn’t help the product, why are you doing it in the first place?

(potentially) related posts:

1. More on the Coding Tester
2. Yet another future of testing post (YAFOTP)
3. Testing with code

I’ll be giving a keynote on testing the Xbox. While I’m certain I’ll have interesting stories and pragmatic examples I can share in a generic fashion, I’m hoping I can share much, much more. I don’t have clearance yet to share too much about the Xbox One, but assuming I can get my proposals in place, it should be a pretty exciting talk to share. By the time STAR rolls around, I’ll have passed my 20 year anniversary working on software, and working on this product has definitely been the highlight of my testing career.

I’m also signed up to give a half-day workshop on testing. Literally – “on testing”. The title of the workshop is Alan Page: On Testing, and I’ll take the time to share some good stories and examples of new testing ideas, but I expect the attendees to drive the content. More details on that as we get closer to the event, but I expect it will be a ton of fun, and especially interesting to those folks who just want to suck up some new and interesting ideas.

More as the conference gets closer.

(potentially) related posts:

1. Twinkle Twinkle I’m back from STAR
2. Happy Birthday HWTSAM
3. Stop Guessing about my STAR presentation

Yesterday, we had our fifth annual “soapbox” event, a lightning talk-ish event where speakers are encouraged to share a rant or opinion with the rest of the community (five minute limit). Because I’ve been thinking so much about my Tear Down the Wall post, I decided to give the five minute version for my peers.

I began with the story of Lloyd Frink (from this post), and  compared the world of technology in 1979 to the world today. I talked about how “walls” get in the way, and how boxing people into roles (especially roles of gatekeeper and fake-customer) are fruitless at best. I closed by sharing Trish Khoo’s quote from her take on all of this:

It would be somewhat revolutionary, if it weren’t already happening.

This was a point I really wanted to drive home. Many of the senior testers at Microsoft have only been at Microsoft, and few pay any attention to software development outside the Borg, and I think there’s a lot to learn from companies and teams that have successfully blurred and erased lines and boxes from their development teams.

As of today, I’m still employed, so I’ll keep ranting and see what happens.

For MS folks, you can find the talk on //resnet (search for soapbox). The whole event is good, but my talk starts about 19:30 in.

(potentially) related posts:

1. Why?
2. Fall Travel
3. The Ballad of the Senior Tester

A few things happened recently to make everything really click for me. One, was this article, asking for universities to offer software testing degrees – as a means to “train our testers”. Besides the fact that the author thinks that Universities are vocational schools where we train people (rather than teach people), I think it’s difficult to have a degree in something that the industry can’t define consistently (yes, I know there are test degree programs out there now, and I could make the same argument about those).

Then, just over a week ago at Telerik’s testing summit, we led off the open-space session with a discussion on test design. I shared some of my ideas, including how I use mind-maps to communicate test design. Towards the end of the discussion, Jeff Morgan (@chzy) asked, “Isn’t it just ‘Design’”. What Jeff was implying was that many of the details of test design should be equally at home when thinking of feature design. Rather than have programmers determine how it should work, and then have testers determine how it should be validated and explored, that those tasks could occur (for the most part) simultaneously – e.g. “To implement this feature, I need to read a value from the database, but I also need to make sure I account for and understand performance implications, whether or not the values will be localized, what the error cases are, etc.).” Much (not all) of test design can be considered when designing code – so why don’t we consider test design and code design simultaneously?

After this discussion (and for many, many hours since telsum), I’ve been thinking about programming and testing and how the two roles can work together better – and began to wonder if the test role may be detrimental to software development.

Three things are important to note before you scroll down and leave me hate-comments.

1. I didn’t say test is dead
2. I said “role”, not activity
3. I purposely used the weasel-word “may” – that statement won’t always be true

And now for some much-needed elaboration.

The Wall

There’s a wall between testers and programmers. Programmers “build”, testers “break” – but passing features back and forth between one group and another is wasteful. It takes time and extra communication to traverse the wall. I spent at least the first few years of my career on one side of the wall catching the crappy code programmers threw to me. Then, I found unit-level bugs and threw those bugs back over the wall to the programmers. Then, we played this wasteful game of catch until it was time to ship. Recently, I saw a fairly prominent tester mention that their bread and butter was participating in a game of bad-code and happy-path-bug crap-catch. While I’m happy that there are easy ways to make money in testing, I’d rather tear my eyelids off than do work like this. I like what I do because it’s hard. Finding happy path bugs in crappy software isn’t hard. It’s boring and practically demeaning.

The Neotsys blog latched onto one of my recent posts in their testing roundup. They said that my team at Microsoft has moved to the “whole-team” approach, but that’s not true. We still have testers and programmer roles – and while programmers frequently write test code and testers frequently write product code there’s still a wall. Programmers are still responsible for writing product code, and testers are still responsible for testing – we just don’t have a problem crossing those lines.

We still have a wall. It’s a small wall, but it’s still there.

Tear Down the Wall

If you played with my recent thought experiments of programmers who can test and testers who can program, it’s not much of a reach to picture a team where there are no programmers or testers – just contributors (or developers or engineers, or whatever you want to call them). If you have a hard time imagining that, imagine your current team with no walls between role responsibilities. Many of you may do the same thing you’re doing today, but with the walls gone, I bet you could make better software – faster.

Figure out what needs to get done, and get it done. Leverage the diversity of the team. If someone is a specialist, let them specialize. If they’re not, let them optimize for their own skills.

Tear down the wall.

Titles

On a quick side note, testers worry (far too much) about their titles. This recent blog reminds me of the idiocy of tester titles (and I’ve discussed it here). There will always be a place for people who know testing to be valuable contributors to software development – but perhaps it’s time for all testing titles to go away?

The Future?

I’ve written in the past about where test is going – toward data analysis, non-functional emphasis, etc., but I think I was at least partially wrong. What software teams need in the future is team members who can perform the activities of programming, testing, and analysis – together.

I saw a slide deck recently that stated, “Agile Testers frequently can’t keep up with Programmers”. Good software development can’t happen when you serialize tasks – the team needs to work on this stuff – together.

I’ll always be deeply involved in the activity of software testing, but I don’t know if the role or title exists in my future. After nearly 20 years of being a tester (and likely several more with that title), I’m admittedly going out on a bit of a limb with that statement. Despite my title, I want the walls to go away.

Testing can’t be something that happens after programming is complete any longer. Testers aren’t “breakers” any more – but experts in the testing activity are (or need to be) as critical to making software as anyone else on the team.

Since it’s mostly testers who read this blog I challenge all of us to shed any remnants of tester vs. programmer and figure out how to tear down the wall.

I’ll let you know how my own battle goes.

(potentially) related posts:

1. What is Testing?
2. Activities and Roles
3. Yet another future of testing post (YAFOTP)

OK – I know you didn’t read it (or you read it already), so I’ll give you the summary. There are coders, testers, coders who test, testers who code, and many variations in between. If it’s good for a coder to have some test knowledge, it certainly can’t hurt to have some knowledge of code if you’re a tester. My contrived exercises asked you to consider what mix of testing and coding knowledge you’d pick when staffing a team. The spectrum of awareness and infusion of coding and testing skills looks a bit like this:

But that’s not (as some of you pointed out) the full picture. I know fantastic testers with tons of experience and the ability to find the the most important issues quickly and efficiently – but they’re jerks (and that’s putting it nicely). It doesn’t matter how awesome you are, if you’re a horrible teammate, I don’t care.

it’s been 35 years or so since I played any paper based role playing games, but when I did, we had these “character sheets” where we kept track of our characters stats and experience. Something like that would give a lot better picture of the full picture of a good software developer.

It’s not perfect, but it helps to see the big picture (and I’ll be the first to admit, it’s difficult, if not impossible, to measure people on a scale like this) – but as a leader, you can certainly consider the type of people you want on your team. You can’t just hire the first code-monkey (or “breaker”) you find and expect to have all of your problems solved. When I make hiring decisions, those decisions are based an itty-bitty bit on whether the candidate can do the job, quite a bit more on how they will fit in with the team, and a whole lot on how well I think they’ll be able to help the team months and years down the line.

And with that, the intro to this blog post has turned into an interlude, so I’ll wrap this up later this week. Great discussion so far (in comments, twitter, and email) – please keep it up.

(potentially) related posts:

1. More on the Coding Tester
2. The Ballad of the Senior Tester
3. My job as a Tester

In the beginning…

In How We Test Software at Microsoft, we told the story of Microsoft hiring our first tester, Lloyd Frink. Lloyd was a high school intern, and his job was to “test” the Basic compiler by running some programs through the compiler and make sure they worked. That was 1979, but full time testers didn’t show up at Microsoft until 1983. I have no recollection or knowledge of what was happening in the rest of the software industry at that time, but I imagine it was probably the same. Since then, Microsoft has hired thousands of testers (we currently employ over 9000), and the industry has hired a kazillion (yes, a made up number, because I really have no way of guessing) software testers (regardless of title).

Now, in the last several years, many teams (including some inside of Microsoft), have moved to the “whole team approach” – where separation of testing and programming roles are blurred or gone (the activities will always exist). Even on teams where separate roles still exist, there is still a significant amount of blurring between which activities each role does as part of their job (for example, I, like many of my test colleagues, make changes to product code, just as many programmers on the team create and edit test code).

The Wall

The metaphorical “wall” between programming and testing is often quite high (as in programmers write code, then throw it over the wall to testers; who, in turn, throw the bugs back over the wall to developers, who make fixes and start the wall-chucking process over again). Lloyd was hired to catch stuff thrown to him over the wall, and I think it’s a fair argument to say it would have been more efficient for developers to run those tests as part of a check-in suite (assuming, of course, they had such a thing).

To be completely transparent, a wall on my team still exists – it’s just a much shorter wall. There are still expectations of programmers and testers that fit into the typical roles, but collaboration and code throwing don’t occur often.

Infection

Elisabeth Hendrickson speaks often of being “test-infected” to describe programmers who “get” testing (I believe the term was coined by Erich Gamma). I for one, can’t think of a good reason for a programmer to not have some test ideas in their head when they write code.

Similarly, I think that testers can benefit from being a bit “code-infected”. Before the self-preserving screams of oppression begin, let me note that I said testers “can benefit” from some code knowledge. I’m not talking about language expertise – a basic understanding of programming logic and some ideas on how computer programs may improve testing is usually plenty. It’s not necessary to know anything about programming to be a tester – I’m just saying it can help.

Beyond the disclaimer, it’s worth elaborating on the last two paragraphs here. I have had testers tell me that knowledge of code will adversely affect the “tester mindset” – that by knowing how to code, the tester will somehow utilize less of their critical thinking or problem solving skills. While I suspect this to be a false assumption, I have no proof of whether this is true or not. But I will hypothesize that if this is true (that code-infection adversely affects testing skill), then it’s just as likely that test-infection adversely effects programming ability. When I look at it this way, the answer clears up a bit. Take a moment to think about the best programmer you know. I’m fortunate enough to work with some really incredible (and some famous) programmers. Every single one of them is test-infected. Yes – I have no hard data, and zero empirical research, so I could be high as a kite on this, but I’d be curious to hear if anyone knows a great programmer who doesn’t have the ability to think at least a little bit like a tester.

Code Infected

So – let’s suppose for a few minutes that code-infection is beneficial for testers. Not even necessarily equally beneficial – just helpful. Is there such a thing as too infected? An old colleague of mine, John, once said,

“I’m not the best tester on the team, and I’m not the best programmer. But, of the programmers on the team, I’m the best tester, and of the testers, I’m the best programmer”.

Which role should John be in? On your team, would he be a programmer, or a tester? Is it a clear cut decision, does it depend, or does it even matter? Is John a test-infected developer, or a code-infected tester? Does it matter?

I think the answer to the above questions are either “It depends”, or “It doesn’t matter”, so let me ask a different question. Do you want a team full of John’s? Maybe you do, or maybe you don’t (or maybe it depends), so let me rephrase yet again in the form of a quick thought experiment.

An experiment

You are forming a nine-person team to make a super cool software application. You get to build a dream team to put it together. On a scale from 1-9, where a 1 is a tester with zero knowledge of programming, a 9 is a developer who can’t even pronounce “unit test”, and a programmer / tester like John is a 5 – which employees do you hire (note that all of these people get paid the same, work the same hours, get along with each other, etc.).

To be fair, I don’t think there’s a “right” answer to this question, but we can certainly explore some options of what would be more or less “right”. One option is to (conveniently) select one of each type. Then you get infection-diversity! However, I’m betting that most of you are thinking, “There’s no room on my team for developers who don’t write unit tests!” So, is “some” testing knowledge enough? Is it possible to have too much? If I hire 7’s and 8’s, do I eventually want them to have more testing knowledge? If that’s the case, should I just hire 5’s and 6s instead?

And what about testers? Remember, that code-infected doesn’t necessarily mean that the testers are code-monkeys cranking out tools and automation all day long – it just means that they have (varying levels of) programming skills as part of their available toolbox. The context of the product may have some influence on my choice, but if you agree that some programming knowledge can’t hurt, then maybe a 2 is better than a 1. But is a 3 better than a 2? “I suppose that “better” depends, but to me, it’s a worthwhile thought process.

But that’s not right…

If you’ve read this far without jumping down to comments to yell at me, let me admit now that I’ve led you in a bad direction. The exercise above sort of assumes that as you gain testing ability, you lose programming ability (and vice-versa). That’s not true in the real world, so let’s try a bit of a twist and see if we can be slightly more realistic. For this exercise, you have the $1,000,000 of salary to spend on a maximum of 10 employees. You have the following additional requirements:

• An employee can have a “programmer” rating of 0-10, where 0 is can’t even write a batch file to 10 where they dream in algorithms.
• Similarly, an employee has a “tester” rating of 0-10 where 0 is no knowledge of testing, and is at least the best tester you’ve ever dreamt of.
• Minimum salary is $50k, and the maximum salary is (arbitrarily) $160,000.

Because context is important, let’s say the app is a social networking site with clients for phones and tablets, as well as a web interface. Your exercise (if you want to play) is to think about the type of people you would use to staff this team. One option, for example, is to have five developers, each with a 10 rating (5x$100,000), paired with 10 testers who also each have a 10 rating (5x$100,000). Another option would be to have 6 developers, each with a 15 rating (e.g. 9-dev and 6-test for  6x$150,000=$800,000) paired with 4 testers, each with a 5 rating (4x$50,000-$20,000). The choice is yours, and there’s no right answer – but I think the thought exercise is interesting.

I’ll close this post by saying that I really, really hate sequels and series when it comes to blog posts (and I’ll add an extra “really” to that statement when referring to my own posts). But there’s more to come here – both in my thoughts on testing and programming, and in more thought exercises that I hope you will consider.

More to come.

(potentially) related posts:

1. Exploring Test Roles
2. R-E-S-P-E-C-T
3. Peering into the white box

1. I’m busy – probably the hardest I’ve worked in all of my time in software. And although there have been a few late nights, the busy isn’t coming from 80-100 hour weeks, it’s coming from 50 hour weeks of working on really hard shit. As a result, I haven’t felt like writing much lately.
2. I can’t tell you.
3. I’m not really doing much testing.

It’s the third point that I want to elaborate on, because it’s sort of true, and sort of not true – but worth discussing / dumping.

First off, I’ve been writing a ton of code recently. But not really much code to test the product directly. Instead, I’ve been neck deep in writing code to help us test the product. This includes both infrastructure (stories coming someday), and tools that help other testers find bugs. I frequently say that a good set of analysis tools to run alongside your tests is like having personal testing assistants. Except you don’t have to pay them, and they don’t usually interrupt you while you’re thinking.

I’ve also been spending a lot of time thinking about reliability, and different ways to measure and report software reliability. There’s nothing really new there other than applying the context of my current project, but it’s interesting work. On top of thinking about reliability and the baggage that goes along with it, I spend a lot of time making sure the right activities to improve reliability happen across the org. I know that some testers identify themselves as “information providers”, but I’ve always found that too passive of a role for my context. My role (and the role of many of my peers) is to not only figure out what’s going on, but to figure out what changes are needed, and then make them happen.

And this last bit is really what I’ve done for years (with different flavors and variations). I find holes and I make sure they get filled. Sometimes I fill the holes. Often, I need to get others to fill the holes – and ideally, make them want to fill them for me. I work hard at this, and while I don’t always succeed, I often do, and I enjoy the work. Most often, I’m driving changes in testing – improving (or changing) test design, test tools, or test strategy. Lately, there’s been a mix of those along with a lot of multi-discipline work. The fuzzy blur between disciplines on our team (and on many teams at MS these days) contributes a lot to that, and just “doing what needs to be done” fills in the rest.

I’m still a tester, of course, and I’ll probably wear that hat until I retire. What I do while wearing that hat will, of course, change often – and that’s (still) ok.

(potentially) related posts:

1. Yet another future of testing post (YAFOTP)
2. The Skeptics Dilemma
3. Activities and Roles

foo.service doesn’t appear to be working

Repro:

1. I installed the build from <build_path>
2. I verified the binaries existed <where they should exist>
3. I queried to see if foo.service was running – it wasn’t
4. I queried another way, and it didn’t show anything either
5. When I run my tests, they fail and tell me that the service isn’t running
6. I thought the service wasn’t started, so I tried starting it, but that also failed (error text: Failed to retrieve the fizzbaz from the bogatorium)

On first glance, it looks like there’s a real problem here. I try to avoid the “works on my machine” comments, but I thought it was strange that nobody else had seen this. I assumed the Problem was Between the Keyboard And the Chair (PBKAC).

At first, that seemed to be exactly the problem. You see, there is no foo.service. It’s actually called fizz.service, but it’s run as part of the foo toolset, so it’s an easy misunderstanding. If they would have queried for fizz.service, they would have seen it happily running.

Their tests failed because they had an invalid command line for their test (or more specifically, they specified an invalid address for the machine where they wanted to run the tests).

And then they got that strange error message attempting to start the service because they were attempting to restart a service that was already running (interesting, however, was that they did manage to try and start the fizz service rather than the foo service at this point.

Three errors, all different, yet relatively easy to see as symptoms of a common problem. Except they weren’t.

Absolutely, definitely, beyond a shadow of a doubt, PBKAC.

But maybe not. Actually, definitely not. The princess is in another castle – or between a different keyboard and chair.

PBKAC was definitely in play when the user tried to query for the wrong service. My service names above are silly, but our service names are actually nearly as confusing. You only make this mistake once, but it’s not too difficult to make.

When the tests failed, the error said the service wasn’t running. The error was 100% accurate (the service wasn’t running because the user connected to the wrong machine). What if, as a courtesy, that error message said something like, “The service isn’t running. Are you connected to a valid target machine”? I bet that would have set off a few light bulbs rather than generated more confusion.

That message about the fizzabaz and the bogatorium isn’t far from what the user actually saw. What they saw was again, a 100% accurate statement of what went wrong – it just gave no clue of what may have caused it.

Granted, these were internal tools, but that’s a horrible excuse to confuse someone. I bet I’ve seen hundreds (if not thousands) of error messages like this – but very few that offer even the tiniest bit of trouble shooting or diagnostic advice. It’s easy to blame the user when they do something “wrong” (or unexpected), but ultimately, it’s rarely their fault. And in the end, if they can’t do what they want to do with your software, they’ll take their business and money elsewhere.

And then, it’s your problem alone.

(potentially) related posts:

1. Settling on Quality?
2. Can you get me a repro?
3. Ur doin it rong

This week, in response to that post (some responses take longer than others), I received a pointer to a little multitasking test that I thought would be fun to share. I’m not affiliated, no kickback, blah blah blah, just sharing because I think it’s interesting. I embedded the test below, but you can go straight to the source at http://open-site.org/blog/the-multitask-test/ as well. (note – the embedded version doesn’t always work that well, so head on over to the open-site.org page if you’re having problems).

(potentially) related posts:

1. Stupid Multitasking
2. Flashes of (near) fame
3. stack exchange, software testing, and learning

This is nothing new – it’s part of the software culture. A few years ago, I recorded a few handfuls of these talks (some slightly staged) with some all-star colleagues and we posted them on the (now-defunct) Microsoft tester center site. Given that the talks are getting pretty difficult to find, I thought I’d take a few of them and re-post to youtube. I have no schedule in mind for posting, but I expect I’ll re-post 10-20 or so in the coming months.

Here are two to get things started. The first is me talking with James Rodriguez about code reviews, and the second is Alan Myrvold getting me started with Security Testing.

(potentially) related posts:

1. One Year of Lync
2. Learning to learn
3. stack exchange, software testing, and learning

Somewhere in the tweet insanity, I promised to blog about my opinions on the concept, so here goes.

First off – this isn’t a new gripe of mine. Here’s a tweet from August, 2011 (edited to fix my typo). To be clear, I think all three of the items below are valuable – yet all also seem to be frequently hyped beyond their value. The gorilla especially comes up among testers – many of whom overreact to the value of the video to software testing.

The video shows that it’s possible to miss something right in front of your eyes. The point of the video is to show that it’s possible to miss something that’s right in front of your eyes.

IB makes you confront the illusion of attention.

That’s it.

Knowing about inattentional blindness doesn’t make you better at noticing things, and there’s zero correlation between the observational skills of those who see the gorilla vs. those who don’t. While I agree that it’s critical for testers to know that inattentional blindness exists, it’s a small nugget of information in a pretty big pool of stuff that actually helps testers. Any knowledge work requires knowledge of IB – furthermore, I’d argue that there are plenty of professions where IB is much more critical to know about than in software testing.

For example:

• Lifeguards
• Judges
• TSA Agents
• Script Supervisor (those are the people in charge of ensuring continuity in movies / tv)
• NASCAR drivers

And I’m hard pressed to think of a profession where knowledge of IB isn’t at least equal to that of testing. My garbage man (sanitation engineer) needs to make sure he doesn’t miss any cans, and ensure I’m not throwing away anything illegal. Given that garbage collection is much more repetitive than anything I do from day to day, I’d expect viewing the gorilla video to be standard training material for the guys in the big stinky truck (and for all I know, it is). The gorilla is fascinating – for everyone. There’s no special appeal to testing that I can see.

Of course, beyond the video (which helped the people behind it earn many awards), Christopher Chabris and Daniel Simons also wrote a book called, The Invisible Gorilla which discussed the vides and a ton of other really cool stuff (stuff I find much more interesting than the gorilla video – especially when considered as a whole). I’ve read the book twice, skimmed it several other times, and met the authors briefly after attending a talk they gave a year or so ago.

Some other cool topics covered include:

• The illusion of memory (what you think you remember clearly may not be accurate. At all).
• The illusion of confidence (most people overate their abilities – includes a great story about chess players and chess rankings)
• The illusion of knowledge (you probably don’t know as much as you think you do)

It’s good stuff.

The Gorilla Stunt is good.

It’s just completely over-hyped in much of  the testing community.

So stop hyping the damn gorilla video.

(potentially) related posts:

1. Judgment in Testing
2. It’s a big world (of testing) out there
3. More on the Coding Tester

The recording is here if you want to take a look.

Video streaming by Ustream

(potentially) related posts:

1. See Me…Hear Me
2. Test Innovation
3. Thoughts on Swiss Testing Day

Of course, remote workers all over the internet are completely up in arms whether they work at Yahoo or not. Perhaps they’re afraid that their employer will follow suit (not-likely), or they don’t want to be discovered as yet-another-wfh-miler (slightly more likely), or they feel like their choice of work method is being threatened by the publicity of this choice (most likely).

To be clear, I am completely supportive of working from home. I’m fortunate enough to have an employer and a history of managers who let me work from home – or remotely as needed. In fact, the recent news reminded me that after I worked remotely for two solid weeks in the summer of 2011, I wrote up some thoughts for my (then) manager. Given that there’s nothing confidential in that write up, I’m sharing it unedited below as fodder for discussion.

This summer, I’m spending two weeks working from Toulouse, France. My family came here for a vacation (and to house sit for some friends of friends). I had planned to only stay for about 10 days, but due to a variety of circumstances, I decided to extend my trip and stay with my family for an additional two weeks. I asked Ross if I could work from France for a few weeks and he graciously allowed me to do so.

The logistics

The house we’re staying in has a reasonably fast internet connection as well as an office where the door shuts, so a reasonable workspace wasn’t a problem. I decided that I’d work Redmond hours (I start work between 4:00 and 6:00 pm and work until 2:00 or 3:00am). There was no requirement that I align my work with Redmond time, but it allowed me to spend some time on daytrips with my family during the day before beginning the workday. I was also able to attend a fair number of meetings over Lync.

The experience

I’ve worked from home before and have never had a problem staying focused on work outside of the workplace (I probably learned to excel in this area while writing hwtsam on evenings and weekends). My family (fortunately) “gets” that I’m working even though I’m close by and leaves me alone to concentrate.

One highlight of the experience is that Lync has worked flawlessly. I’ve made several calls to Redmond, and attended several meetings. Audio and video have worked well, and it’s helped keep me connected much of the time.

I was reflecting on my first week of working remotely, and had a bit of an insight. I have been able to get a ton of work done – but to be fair and honest, it’s different work than I would have done had I been in Redmond. I don’t think this is necessarily a bad thing, since some of the things I’ve worked on (e.g. writing up thoughts and experimenting with fault injection, figuring out how Lync should approach model-based testing, writing up debugging tutorials, or polishing up a thinkweek paper) are the right work for me to do, but it seems that working remotely changes priorities slightly. By this, I mean that a big part of my typical role involves interacting with people on the team in a somewhat random pattern – e.g. answering questions in the hallway, discussing topics of the day over lunch, or following up with people 1:1 after meetings. Not all Microsoft roles involve this sort of interaction, but it seems difficult to interact in this manner remotely.

One example I thought of that reflects the above is this scenario: Say Josh and Bob are talking in Josh’s office. I overhear the conversation and have some relevant (and valuable!) thoughts, so I get up, poke my head in and join the conversation. That scenario doesn’t happen if I’m not there.

An interesting Lync feature based on this would work like this. Say Bob and Josh are having an IM conversation. If Lync noticed that I was on both of their contact lists, and they mentioned a keyword that shows up in my “interests”, Lync would ask them if they wanted to add me to the conversation.

I think the casual interaction limitation is more of a cultural problem than something inherent to working remotely (and something that may solve itself if I was away for a longer period). My thought is that in general, people on our team don’t send IM’s casually – e.g. “Hey – I was just talking to Dan about test automation, and wondered if you had thoughts on how to do data driven testing”, or “Do you have any quick thoughts on blah?”. The questions I normally hear in the hallway, or from someone sticking their head in my office haven’t occurred over IM (or telephone for that matter).

Another example is the value (in our culture) of the in-person follow up. For example, I sent an email to a few peers on the team last week – I was expecting it to turn into a discussion, but only received two short replies (I replied to both, and then the conversation ended). If I were in Redmond, I probably would have had an additional casual conversation with a few of the recipients and attempted to clear up any ambiguity or answer any questions that was blocking an engaged conversation on the topic. This is difficult to do remotely (although it’s certainly possible to do all of this over IM, it takes some getting used to).

Now – since I wrote that email, I’ve changed teams, and the world has warmed up more to social media. Lots of Microsoftees use Yammer, and IM usage is on the rise. I (and others) still don’t think we (as a company, and as an industry) are doing enough to support and encourage remote work, and I’m discouraged a bit that Yahoo has seemed to take us a step or two backwards.

(potentially) related posts:

1. Some thoughts on remote work
2. Fuel for work
3. Getting back to work

However – if you are also in the Pacific Northwest, and want to hang out and talk about test innovation, I’ll be hanging out with the folks at qasig on the evening of March 13. I’ll be delivering an updated version of my EuroStar keynote on test innovation, and I expect it will be a fun night.

Beyond that (and depending on interest), I may do another free webinar soon (it’s been at least a year since the last one), but that will be about it for me talking about testing in person about testing.

More here later.

-AW

(potentially) related posts:

1. Upcoming stuff
2. Eurostar 2012 – Slides
3. Surface RT and EuroSTAR recap

“Hey – can you set up a repro of that bug for me?”

As a tester, how many times have you heard this phrase? How many times have you walked through the steps you outlined in the bug report so someone could look at an error for you? Or – how many times have you seen a test error, and immediately re-run the test to see if you could reproduce the error yourself?

Is it a big number? If it is, you’re not going to like what I have to say. If you need to reproduce all of your bugs to figure out what’s going on, you screwed up. Your logging is bad. Your diagnostics don’t exist. You wrote crummy code. You’re wasting time!

I sometimes see code like this:
int status = CreateSomethingCool(object);

if (status != COOL_SUCCESS)

{
    return -1;
}
status = MakeItWayCool(object);
if (status != COOL_SUCCESS)
{
    return -1;
}

The next time the test is run, Joe Tester (assuming some sort of automation around the automation) gets an email saying his test failed. Of course, Joe doesn’t have a freakin’ clue why his test failed. It could have failed for either case above (assuming those are the only two failure points), so he needs to run it – perhaps under a debugger to see what happened.

Joe writes horrible test code.

But – Joe wants to improve, so he writes this:

int status = CreateSomethingCool(object);

if (status != COOL_SUCCESS)
{
    return -1;
}
status = MakeItWayCool(object);

if (status != COOL_SUCCESS)
{
    return -2;
}

Now Joe has different return values, but he still writes crappy code. At least now he knows why his test failed; but he doesn’t know why the product failed.

So, I fired Joe, and hired Sally. Sally wrote this instead.

LOG(“Calling CreateSomethingCool with object = %s”, object.ToString());
int status = CreateSomethingCool(object);

VERIFY_COOLSUCCESS((status), “CreateSomethingCool Failed. status=%d, objectData=%d”, status, DumpObject(object));

LOG(“Calling MakeItWayCool with object == %s”, object.ToString());
status = MakeItWayCool(object);

VERIFY_COOLSUCCESS((status), “CreateSomethingCool Failed. status=%d, objectData=%d”, status, DumpObject(object));

To be fair, Sally’s version was a little easier to read, and contained some comments (or it would if Sally was a real person), but when her test failed, instead of needing to set up a repro, the automation system sent her this mail.

Test Failed

CoolTest failed with a verify failure. Log follows:
Calling CreateSomethingCool with object = LittleRedCorvette
CreateSomethingCool Failed. status=5, objectData=
    Name=LittleRedCorvette
Size=0×100
Active=false
Running=false
LastError=5

Even here, there may not be enough information, but chances are that if Sally (or her teammates know that), “CoolTest is failing with object LittleRedCorvette, and it’s likely because the object was inactive and not running, and that the error code was 5″, that someone familiar with the code would know exactly where to look.

And – in the case where Sally (or a teammate) has to hook up a debugger anyway, they should add additional debug information to the log file for the next time a similar error happens. Setting up a repro wastes time. Doing it for every issue you find is irresponsible and wasteful. Be a professional, and stop setting up repros, and start writing tests (and code) that make your job easier and make your team better.

(potentially) related posts:

1. Stop Guessing about my STAR presentation
2. It’s (probably) a Design Problem
3. GUI Schmooey

Regardless of how you feel about the health of software testing (which depends largely on your ability to interpret a metaphor), for me, it’s getting easier and easier to see that testing is changing. Granted, it’s changing along with the software under test, so if you’re testing the same sort of desktop software products you have been for years, using development practices even older, the good news is that you’re safe – your testing world probably isn’t going to change either.

The rest of us are working on software that releases quickly and often, using development practices that support that cadence. In our world, it just doesn’t make sense for a test team to invest a bunch of time in functional testing. It’s cheaper and more cost efficient to have the programmers who write the code write tests to verify unit and functional correctness. This eliminates unnecessary back and forth, and forces programmers to write more testable (and often simpler) code from the beginning, resulting in easier maintainability and extension of the code.

Of course, you don’t need to tell me that it’s a long leap from functional correctness to usable software. Given a beer or two, I’ll give you names of products I’ve worked on that have been near functionally perfect, yet near failures in the market. This gap is where my future of test lives. Another colleague (one with no blog) says, “We can define what programmers do, and what program managers do fairly easily. Testers do the remainder.” This statement remains correct – even if “the remainder” is a moving target.

One big role falling into the remainder is that of data analysis / data science / data interpretation / whatever you want to call the analysis of customer data rolling in. As products move more and more into the cloud, there are more and more opportunities to run tests and analysis in production and get data in near real time. I honestly think that the ability to provide actionable product insights from terabytes or more of data is the key to a six-figure plus paycheck for decades to come. Some testers will fit naturally into this role – but I have a hunch we’ll find a lot of people in this role with backgrounds in Mathematics or Statistics than from Computer Science.

When you think about “the remainder”, there’s another big hole. I think we’ll always need people to look at big end-to-end scenarios and determine how non-functional attributes (e.g. performance, privacy, usability, reliability, etc.) contribute to the user experience. Some of this evaluation will come from manually walking through scenarios), but there will be plenty of need for programmatic measurement and analysis as well (e.g. is there real value in manual performance tests, or manual stress tests?). I don’t know if there’d be more or less specialization than there is today, and don’t know if it matters…but it may.

There may be other new roles, while some roles abundant today may go away – although not immediately, as I still see several openings for “Functional Test Engineer” on a popular job site. Short story is that I’m cool with this future. Others may not be, and that’s ok. I’m just happy to ride the wave.

(potentially) related posts:

1. What is Testing?
2. New Testing Ideas
3. Forward thinking in software testing

In fact, I’d say there’s a minimum amount of information that every tester should know about debuggers, regardless of whether they are monster-rock-star programmers, or part-time IT application validators. I’ll start a list here, and likely add to it when I’m annoyed.

So here goes:

Things Every Tester Should Know About Debuggers

• Call Stacks – A “call stack” is a list of functions that led up to the application crash. Different debuggers display this differently, but it will look something like this:

  FunctionThatCrashed
  Function_Three
  Function_Two
  Function_One

  Read this from bottom to top – Function_One called Function_Two, which called Function_Three, which called FunctionThatCrashed (which crashed).

  Typically you’ll see parameters after the functions, and those will probably give you a clue. If you’re working with developers, and you see a crash in the debugger, the call stack is probably the most important contextual information you, as a tester, can provide.

• Crash vs. Assert – When a program breaks into the debugger (meaning the program stops executing, and the debugger springs into action), the program may have crashed (e.g. attempted to read/write to/from invalid memory), or it could have broken at a developer-induced breakpoint (often in a macro called an ‘assert’). On x86 / amd64 platforms this shows up as an int 3
  or int 2C in the debugger. The important thing to note in the case of the assert type of error is that the reason for the break is obvious  – or at least obvious if you can view the code. In practice, asserts often look like this:

      bool bSuccess = CreateUserAccount();
      //break into the debugger of the call to CUA fails so we can debug
      ASSERT(bSuccess == true);

When a tester sees this break, instead of saying, “The program crashed during logon”, they can say, “I’m hitting an assert because CreateUserAccount is failing on today’s build”. While both statements have value, the latter statement is much more informative, and in many cases actually actionable.

• Basic Lingo – Know a little about a few basic types of crashes – e.g. Access Violation (AV), Stack Overflow, or Buffer Overrun, and know a little about what causes these errors (e.g. reading from a bad memory address, calling too many functions – usually via recursion, or putting too long of a string into a buffer). You don’t have to be an expert, but a little knowledge here goes a long, long way in diagnosing bugs and getting them fixed.
  

Off the top of my head, that seems like enough, but not that much. There’s certainly more to learn, but I think knowing just this much should be a minimum for any professional tester.

More ideas – add ‘em to the comments.

(potentially) related posts:

1. Titles for Testers
2. Online Presentation: Career Tips for Testers
3. Career Paths for Testers Using Personas

One common example from my life is the mediocre musician who decides to get better by taking lessons from a master. This isn’t a bad choice by itself, but in many cases, months go by, and while they may make some minor improvements, there are no dramatic (or often, noticeable) improvements. Meanwhile, another mediocre musician makes huge advancements without a single lesson. The difference is that the latter musician invested in deliberate practice, while the former assumed the lessons were enough. The joke about the way to get to Carnegie Hall applies well here.

The same thing applies outside of music as well. Taking a skating lesson will help you with basics, but you need to practice to improve those techniques. You need to learn what works for you, and how motions feel, and need to get confidence in your abilities until you hit a point where additional lessons or training may help you make another leap forward.

Of course, this also applies to software engineering practices. Whether you’ve taken a class, or read a book (or even a Wikipedia article), you don’t really know the practice until you’ve tried it yourself (you certainly wouldn’t give a skating lesson or write an article on learning how to skate after an introductory lesson). For anyone interested in improving or learning, you are responsible for finding approaches you haven’t used before (not just those you haven’t read about before), and using deliberate practice to discover the details about what works, and does not work for your context.

I’m not saying practice alone is enough, but you need a balance (note, “Balance” does not necessarily mean 50-50) between deliberate practice and increasing your capacity to learn. One way to increase your capacity is to take a lesson or a training course. As I mentioned, you can also read about something new. For musicians, playing with other musicians is a fantastic way to learn and increase capacity – this is why I recommend pairing and coaching in software engineering so much – it’s a great way for all parties to learn and increase their ability to take advantage of deliberate practice.

(potentially) related posts:

1. Improvement through practice
2. Learning to learn
3. Trivial Pursuits

Posts related to software forthcoming…HNY.

(potentially) related posts:

1. 2011 year end roundup
2. HWTSAM – One Year Later
3. Another year bites the dust

Some more from the non-Microsoft front follows.

My most popular blog posts from the last year are (were):

• Musings on Test Design
• Orchestrating Test Automation
• Exploring Test Roles
• Exploring Test Automation
• Quick Testing Challenge
• New Testing Ideas

As I look at those topics, something tells me I should write about testing more, and skip non-testing topics (like year-end recaps, for example).

I spent most of the year close to home, hanging out with my family, and focusing on my job. I’ve been on the Xbox team for just over a year now, and I’m still having a great time. It’s a fantastic place to work, and I’m surrounded by more brilliant hard-working and passionate people than you could imagine. I will see if I can share some more stories and insights about the team in 2013.

I was visiting my parents this summer when my mother was diagnosed with cancer. I had committed to three speaking engagements in the fall, and in between those, work, and heading to my parents’ house every weekend to help out, late summer and fall were exhausting.

I was honored to be invited to be part of the STANZ tour of New Zealand and Australia in September put on by SoftEd. I can’t say enough good things about the SofEd staff – they took care of every detail before, during, and after the event. It was a fantastic experience, and I hope I can get myself invited again some day. I had planned to take an extra few weeks off to travel after the conference, but given my mom’s health, I headed back early.

In October I gave a talk (that I thought wend quite well) at Intel in Oregon. This is my second time speaking at Intel (I spoke in Israel last year), and would speak there any time.

My next speaking event wasn’t planned. I was at my mom’s side when she passed later in October, and I gave the eulogy at her funeral a few days later.

A week later, I was on a plane to Amsterdam to give the opening keynote at EuroStar (slides are in my previous post). The staff and program committee at EuroStar were well-organized, and the conference ran (from my view, at least) quite smoothly. In the end, I’m sorry to say that I delivered a flat (by my standards) presentation. I was thrilled that some people said they liked it, but I wasn’t happy with my delivery. Given everything going on at home, I flew home before the conference ended and began trying to get back into a more normal routine.

The “normal routine” for me these days is work, family, and when I can find time, a bit of Xbox gaming (gamertag: A Weezil). I may start blogging regularly again in the new year – or I may not – I don’t know yet. I’ve committed to one conference next fall, and may do one other one. And that’s about all I’ve figured out.

I’m thankful for all of the new people I’ve met this year (I met too many testers at conferences this year to list their names), and for people I got to know better. For the first time in years, I have a bit of excitement about where testing is going (although some readers may not like where I think testing is going…). If you’ve read this far, I wish you luck in 2013 (and for the rest of 2012). I hope to cross paths with many more of you in the months to come.

(potentially) related posts:

1. Fall Travel
2. New Zealand & Australia, 2012
3. Oh, the places I’ll go