I frequently go days at work without writing code. On many days, I do “typical” test activities (seeing what works, what doesn’t work, etc.) Some days I’m debugging all day. Other days, I’m sifting through the perl and batch scripts that bolt our build process together. On other days, I’m managing projects, tweaking excel, or trying not to cause too many problems in SQL. This afternoon, at least after lunch, was coding time – and probably the first extended code time I’ve had in a few weeks.

Without too many boring details, I had a backlog item to write a service to automatically launch a memory analysis tool and periodically take snapshots of memory allocations in specified processes. The memory analysis tools (which I also wrote adapted from another project) work really well, but need a lot of manual intervention in order to get accurate data. I wrote a quick “spec” on my whiteboard, wrote tests as I went along, and in about four hours I had it up and running successfully. I’m confident that the service will make it much easier for our team to find and isolate memory issues. To me, writing this tool was a testing activity. It doesn’t test anything, but certainly makes testing of the product much easier.

On my drive home, I was pondering the universe (and my day) and remembered that there’s a sad bit of truth in the software world. It’s a fallacy that causes great confusion. It promotes poor strategies, bad hiring decisions, and endless confusion among management, contributors, and consultants. It may very well be the most misunderstood word in the world of software. It’s the “A” word that is practically ruining testing.

Automation.

Yes – as much as I hate them, I used a one word paragraph. I did it because the abuse and misuse of “automation” in testing is causing too many smart people to do too many dumb things. That said, I suppose an explanation is in order.

Once upon a time, someone noticed that testers who knew how to code (or coders who knew how to test) were doing some wicked-cool testing. They had tools and stress tests and monitoring and analysis and they found great bugs that nobody else could find – “Test Developers” were all the rage…so we (the industry) hired more of them. Not much later (and possibly simultaneously), someone else decided that testers who could code could automate all of the tasks that the silly testers who didn’t code used to do. Instead of finding really cool bugs, coding testers were now automating a bunch of steps previously done manually. Not to be left out, UI automation frameworks began to sprout, with a promise that “anyone can automate – even a tester.” Now, test teams can spend the bulk of their time maintaining and debugging tests that never should have been automated in the first place.

And that was the first step into darkness.

This mindset, although common, kills me:

• Testers who code write automation
• Automation is primarily made up of a suite of user tasks
• Automation is primarily done through the GUI

All of these items are, in my opinion, idiotic – and likely harmful to software quality. It focuses test teams in the wrong place and is, frankly, insulting to testers who know coding.

If you’re a tester who writes code (or a coder who cares about testing), try this:

• Write tools that help you find important bugs quickly
• Write a few high level tests that ensure user actions “work”, but leave most of it for exploratory testing, user testing (assuming you have great monitoring tools), or both.
• If you must write GUI tests for the above, write fewer of them (that way you’ll spend less time investigating their flaky behavior).

If you’re a tester who writes code, you’re not a test automator, you’re not a test automation engineer, and you don’t write test automation or test code.

You write code. And you test.

You write code that helps the product get out the door sooner. And if the code you write (or the testing you do) doesn’t help the product, why are you doing it in the first place?

The cat’s out of the bag – I’m popping out of my no-conference bubble, and making an appearance at a testing conference (STAR West in October). The theme of the conference is “Be a Testing Rock star”, and while I think that theme begs for an appearance from Michael Larsen, I’ll do my best to live up to the hype.

I’ll be giving a keynote on testing the Xbox. While I’m certain I’ll have interesting stories and pragmatic examples I can share in a generic fashion, I’m hoping I can share much, much more. I don’t have clearance yet to share too much about the Xbox One, but assuming I can get my proposals in place, it should be a pretty exciting talk to share. By the time STAR rolls around, I’ll have passed my 20 year anniversary working on software, and working on this product has definitely been the highlight of my testing career.

I’m also signed up to give a half-day workshop on testing. Literally – “on testing”. The title of the workshop is Alan Page: On Testing, and I’ll take the time to share some good stories and examples of new testing ideas, but I expect the attendees to drive the content. More details on that as we get closer to the event, but I expect it will be a ton of fun, and especially interesting to those folks who just want to suck up some new and interesting ideas.

More as the conference gets closer.

I chair an advisory council for a community of senior testers at Microsoft. We have a variety of events ranging from talking heads to open space events to panels to whatever type of event we think is the most different than the previous one.

Yesterday, we had our fifth annual “soapbox” event, a lightning talk-ish event where speakers are encouraged to share a rant or opinion with the rest of the community (five minute limit). Because I’ve been thinking so much about my Tear Down the Wall post, I decided to give the five minute version for my peers.

I began with the story of Lloyd Frink (from this post), and  compared the world of technology in 1979 to the world today. I talked about how “walls” get in the way, and how boxing people into roles (especially roles of gatekeeper and fake-customer) are fruitless at best. I closed by sharing Trish Khoo’s quote from her take on all of this:

It would be somewhat revolutionary, if it weren’t already happening.

This was a point I really wanted to drive home. Many of the senior testers at Microsoft have only been at Microsoft, and few pay any attention to software development outside the Borg, and I think there’s a lot to learn from companies and teams that have successfully blurred and erased lines and boxes from their development teams.

As of today, I’m still employed, so I’ll keep ranting and see what happens.

For MS folks, you can find the talk on //resnet (search for soapbox). The whole event is good, but my talk starts about 19:30 in.

It’s interesting when I go back and look at the number of posts where I talk about what I do, what testing is to me, and how testing is changing. Ever since the Telerik Test Summit (telsum), I’ve been thinking even more about testing and how it fits into software development. When I wrote this post, and added on a bit to it here, I was pondering the same thing. When I (last) blogged about the future of testing, I (likely subconsciously) was thinking about the same thing.

A few things happened recently to make everything really click for me. One, was this article, asking for universities to offer software testing degrees – as a means to “train our testers”. Besides the fact that the author thinks that Universities are vocational schools where we train people (rather than teach people), I think it’s difficult to have a degree in something that the industry can’t define consistently (yes, I know there are test degree programs out there now, and I could make the same argument about those).

Then, just over a week ago at Telerik’s testing summit, we led off the open-space session with a discussion on test design. I shared some of my ideas, including how I use mind-maps to communicate test design. Towards the end of the discussion, Jeff Morgan (@chzy) asked, “Isn’t it just ‘Design’”. What Jeff was implying was that many of the details of test design should be equally at home when thinking of feature design. Rather than have programmers determine how it should work, and then have testers determine how it should be validated and explored, that those tasks could occur (for the most part) simultaneously – e.g. “To implement this feature, I need to read a value from the database, but I also need to make sure I account for and understand performance implications, whether or not the values will be localized, what the error cases are, etc.).” Much (not all) of test design can be considered when designing code – so why don’t we consider test design and code design simultaneously?

After this discussion (and for many, many hours since telsum), I’ve been thinking about programming and testing and how the two roles can work together better – and began to wonder if the test role may be detrimental to software development.

Three things are important to note before you scroll down and leave me hate-comments.

1. I didn’t say test is dead
2. I said “role”, not activity
3. I purposely used the weasel-word “may” – that statement won’t always be true

And now for some much-needed elaboration.

The Wall

There’s a wall between testers and programmers. Programmers “build”, testers “break” – but passing features back and forth between one group and another is wasteful. It takes time and extra communication to traverse the wall. I spent at least the first few years of my career on one side of the wall catching the crappy code programmers threw to me. Then, I found unit-level bugs and threw those bugs back over the wall to the programmers. Then, we played this wasteful game of catch until it was time to ship. Recently, I saw a fairly prominent tester mention that their bread and butter was participating in a game of bad-code and happy-path-bug crap-catch. While I’m happy that there are easy ways to make money in testing, I’d rather tear my eyelids off than do work like this. I like what I do because it’s hard. Finding happy path bugs in crappy software isn’t hard. It’s boring and practically demeaning.

The Neotsys blog latched onto one of my recent posts in their testing roundup. They said that my team at Microsoft has moved to the “whole-team” approach, but that’s not true. We still have testers and programmer roles – and while programmers frequently write test code and testers frequently write product code there’s still a wall. Programmers are still responsible for writing product code, and testers are still responsible for testing – we just don’t have a problem crossing those lines.

We still have a wall. It’s a small wall, but it’s still there.

Tear Down the Wall

If you played with my recent thought experiments of programmers who can test and testers who can program, it’s not much of a reach to picture a team where there are no programmers or testers – just contributors (or developers or engineers, or whatever you want to call them). If you have a hard time imagining that, imagine your current team with no walls between role responsibilities. Many of you may do the same thing you’re doing today, but with the walls gone, I bet you could make better software – faster.

Figure out what needs to get done, and get it done. Leverage the diversity of the team. If someone is a specialist, let them specialize. If they’re not, let them optimize for their own skills.

Tear down the wall.

Titles

On a quick side note, testers worry (far too much) about their titles. This recent blog reminds me of the idiocy of tester titles (and I’ve discussed it here). There will always be a place for people who know testing to be valuable contributors to software development – but perhaps it’s time for all testing titles to go away?

The Future?

I’ve written in the past about where test is going – toward data analysis, non-functional emphasis, etc., but I think I was at least partially wrong. What software teams need in the future is team members who can perform the activities of programming, testing, and analysis – together.

I saw a slide deck recently that stated, “Agile Testers frequently can’t keep up with Programmers”. Good software development can’t happen when you serialize tasks – the team needs to work on this stuff – together.

I’ll always be deeply involved in the activity of software testing, but I don’t know if the role or title exists in my future. After nearly 20 years of being a tester (and likely several more with that title), I’m admittedly going out on a bit of a limb with that statement. Despite my title, I want the walls to go away.

Testing can’t be something that happens after programming is complete any longer. Testers aren’t “breakers” any more – but experts in the testing activity are (or need to be) as critical to making software as anyone else on the team.

Since it’s mostly testers who read this blog I challenge all of us to shed any remnants of tester vs. programmer and figure out how to tear down the wall.

I’ll let you know how my own battle goes.

A few days ago, I began a verbal exploration of testers who code and coders who test. That post provides some context for today’s continuation, so if you have a moment, go ahead and check it out.

OK – I know you didn’t read it (or you read it already), so I’ll give you the summary. There are coders, testers, coders who test, testers who code, and many variations in between. If it’s good for a coder to have some test knowledge, it certainly can’t hurt to have some knowledge of code if you’re a tester. My contrived exercises asked you to consider what mix of testing and coding knowledge you’d pick when staffing a team. The spectrum of awareness and infusion of coding and testing skills looks a bit like this:

But that’s not (as some of you pointed out) the full picture. I know fantastic testers with tons of experience and the ability to find the the most important issues quickly and efficiently – but they’re jerks (and that’s putting it nicely). It doesn’t matter how awesome you are, if you’re a horrible teammate, I don’t care.

it’s been 35 years or so since I played any paper based role playing games, but when I did, we had these “character sheets” where we kept track of our characters stats and experience. Something like that would give a lot better picture of the full picture of a good software developer.

It’s not perfect, but it helps to see the big picture (and I’ll be the first to admit, it’s difficult, if not impossible, to measure people on a scale like this) – but as a leader, you can certainly consider the type of people you want on your team. You can’t just hire the first code-monkey (or “breaker”) you find and expect to have all of your problems solved. When I make hiring decisions, those decisions are based an itty-bitty bit on whether the candidate can do the job, quite a bit more on how they will fit in with the team, and a whole lot on how well I think they’ll be able to help the team months and years down the line.

And with that, the intro to this blog post has turned into an interlude, so I’ll wrap this up later this week. Great discussion so far (in comments, twitter, and email) – please keep it up.

For the last day or so, I’ve been thinking a lot about programming and testing, about collaboration and walls, and about where this may all be going. This post is the start of my mental exploration of the subject.

In the beginning…

In How We Test Software at Microsoft, we told the story of Microsoft hiring our first tester, Lloyd Frink. Lloyd was a high school intern, and his job was to “test” the Basic compiler by running some programs through the compiler and make sure they worked. That was 1979, but full time testers didn’t show up at Microsoft until 1983. I have no recollection or knowledge of what was happening in the rest of the software industry at that time, but I imagine it was probably the same. Since then, Microsoft has hired thousands of testers (we currently employ over 9000), and the industry has hired a kazillion (yes, a made up number, because I really have no way of guessing) software testers (regardless of title).

Now, in the last several years, many teams (including some inside of Microsoft), have moved to the “whole team approach” – where separation of testing and programming roles are blurred or gone (the activities will always exist). Even on teams where separate roles still exist, there is still a significant amount of blurring between which activities each role does as part of their job (for example, I, like many of my test colleagues, make changes to product code, just as many programmers on the team create and edit test code).

The Wall

The metaphorical “wall” between programming and testing is often quite high (as in programmers write code, then throw it over the wall to testers; who, in turn, throw the bugs back over the wall to developers, who make fixes and start the wall-chucking process over again). Lloyd was hired to catch stuff thrown to him over the wall, and I think it’s a fair argument to say it would have been more efficient for developers to run those tests as part of a check-in suite (assuming, of course, they had such a thing).

To be completely transparent, a wall on my team still exists – it’s just a much shorter wall. There are still expectations of programmers and testers that fit into the typical roles, but collaboration and code throwing don’t occur often.

Infection

Elisabeth Hendrickson speaks often of being “test-infected” to describe programmers who “get” testing (I believe the term was coined by Erich Gamma). I for one, can’t think of a good reason for a programmer to not have some test ideas in their head when they write code.

Similarly, I think that testers can benefit from being a bit “code-infected”. Before the self-preserving screams of oppression begin, let me note that I said testers “can benefit” from some code knowledge. I’m not talking about language expertise – a basic understanding of programming logic and some ideas on how computer programs may improve testing is usually plenty. It’s not necessary to know anything about programming to be a tester – I’m just saying it can help.

Beyond the disclaimer, it’s worth elaborating on the last two paragraphs here. I have had testers tell me that knowledge of code will adversely affect the “tester mindset” – that by knowing how to code, the tester will somehow utilize less of their critical thinking or problem solving skills. While I suspect this to be a false assumption, I have no proof of whether this is true or not. But I will hypothesize that if this is true (that code-infection adversely affects testing skill), then it’s just as likely that test-infection adversely effects programming ability. When I look at it this way, the answer clears up a bit. Take a moment to think about the best programmer you know. I’m fortunate enough to work with some really incredible (and some famous) programmers. Every single one of them is test-infected. Yes – I have no hard data, and zero empirical research, so I could be high as a kite on this, but I’d be curious to hear if anyone knows a great programmer who doesn’t have the ability to think at least a little bit like a tester.

Code Infected

So – let’s suppose for a few minutes that code-infection is beneficial for testers. Not even necessarily equally beneficial – just helpful. Is there such a thing as too infected? An old colleague of mine, John, once said,

“I’m not the best tester on the team, and I’m not the best programmer. But, of the programmers on the team, I’m the best tester, and of the testers, I’m the best programmer”.

Which role should John be in? On your team, would he be a programmer, or a tester? Is it a clear cut decision, does it depend, or does it even matter? Is John a test-infected developer, or a code-infected tester? Does it matter?

I think the answer to the above questions are either “It depends”, or “It doesn’t matter”, so let me ask a different question. Do you want a team full of John’s? Maybe you do, or maybe you don’t (or maybe it depends), so let me rephrase yet again in the form of a quick thought experiment.

An experiment

You are forming a nine-person team to make a super cool software application. You get to build a dream team to put it together. On a scale from 1-9, where a 1 is a tester with zero knowledge of programming, a 9 is a developer who can’t even pronounce “unit test”, and a programmer / tester like John is a 5 – which employees do you hire (note that all of these people get paid the same, work the same hours, get along with each other, etc.).

 

To be fair, I don’t think there’s a “right” answer to this question, but we can certainly explore some options of what would be more or less “right”. One option is to (conveniently) select one of each type. Then you get infection-diversity! However, I’m betting that most of you are thinking, “There’s no room on my team for developers who don’t write unit tests!” So, is “some” testing knowledge enough? Is it possible to have too much? If I hire 7’s and 8’s, do I eventually want them to have more testing knowledge? If that’s the case, should I just hire 5’s and 6s instead?

And what about testers? Remember, that code-infected doesn’t necessarily mean that the testers are code-monkeys cranking out tools and automation all day long – it just means that they have (varying levels of) programming skills as part of their available toolbox. The context of the product may have some influence on my choice, but if you agree that some programming knowledge can’t hurt, then maybe a 2 is better than a 1. But is a 3 better than a 2? “I suppose that “better” depends, but to me, it’s a worthwhile thought process.

But that’s not right…

If you’ve read this far without jumping down to comments to yell at me, let me admit now that I’ve led you in a bad direction. The exercise above sort of assumes that as you gain testing ability, you lose programming ability (and vice-versa). That’s not true in the real world, so let’s try a bit of a twist and see if we can be slightly more realistic. For this exercise, you have the $1,000,000 of salary to spend on a maximum of 10 employees. You have the following additional requirements:

• An employee can have a “programmer” rating of 0-10, where 0 is can’t even write a batch file to 10 where they dream in algorithms.
• Similarly, an employee has a “tester” rating of 0-10 where 0 is no knowledge of testing, and is at least the best tester you’ve ever dreamt of.
• Minimum salary is $50k, and the maximum salary is (arbitrarily) $160,000.

Because context is important, let’s say the app is a social networking site with clients for phones and tablets, as well as a web interface. Your exercise (if you want to play) is to think about the type of people you would use to staff this team. One option, for example, is to have five developers, each with a 10 rating (5x$100,000), paired with 10 testers who also each have a 10 rating (5x$100,000). Another option would be to have 6 developers, each with a 15 rating (e.g. 9-dev and 6-test for  6x$150,000=$800,000) paired with 4 testers, each with a 5 rating (4x$50,000-$20,000). The choice is yours, and there’s no right answer – but I think the thought exercise is interesting.

I’ll close this post by saying that I really, really hate sequels and series when it comes to blog posts (and I’ll add an extra “really” to that statement when referring to my own posts). But there’s more to come here – both in my thoughts on testing and programming, and in more thought exercises that I hope you will consider.

More to come.

I haven’t blogged much recently, and it’s mainly for three reasons.

1. I’m busy – probably the hardest I’ve worked in all of my time in software. And although there have been a few late nights, the busy isn’t coming from 80-100 hour weeks, it’s coming from 50 hour weeks of working on really hard shit. As a result, I haven’t felt like writing much lately.
2. I can’t tell you.
3. I’m not really doing much testing.

It’s the third point that I want to elaborate on, because it’s sort of true, and sort of not true – but worth discussing / dumping.

First off, I’ve been writing a ton of code recently. But not really much code to test the product directly. Instead, I’ve been neck deep in writing code to help us test the product. This includes both infrastructure (stories coming someday), and tools that help other testers find bugs. I frequently say that a good set of analysis tools to run alongside your tests is like having personal testing assistants. Except you don’t have to pay them, and they don’t usually interrupt you while you’re thinking.

I’ve also been spending a lot of time thinking about reliability, and different ways to measure and report software reliability. There’s nothing really new there other than applying the context of my current project, but it’s interesting work. On top of thinking about reliability and the baggage that goes along with it, I spend a lot of time making sure the right activities to improve reliability happen across the org. I know that some testers identify themselves as “information providers”, but I’ve always found that too passive of a role for my context. My role (and the role of many of my peers) is to not only figure out what’s going on, but to figure out what changes are needed, and then make them happen.

And this last bit is really what I’ve done for years (with different flavors and variations). I find holes and I make sure they get filled. Sometimes I fill the holes. Often, I need to get others to fill the holes – and ideally, make them want to fill them for me. I work hard at this, and while I don’t always succeed, I often do, and I enjoy the work. Most often, I’m driving changes in testing – improving (or changing) test design, test tools, or test strategy. Lately, there’s been a mix of those along with a lot of multi-discipline work. The fuzzy blur between disciplines on our team (and on many teams at MS these days) contributes a lot to that, and just “doing what needs to be done” fills in the rest.

I’m still a tester, of course, and I’ll probably wear that hat until I retire. What I do while wearing that hat will, of course, change often – and that’s (still) ok.

Yesterday, I read a mail sent to an email alias I’m on, where the author was asking why tool X wasn’t enabled on his latest build. The mail looked something like this (genericized to protect the innocent).

foo.service doesn’t appear to be working

Repro:

1. I installed the build from <build_path>
2. I verified the binaries existed <where they should exist>
3. I queried to see if foo.service was running – it wasn’t
4. I queried another way, and it didn’t show anything either
5. When I run my tests, they fail and tell me that the service isn’t running
6. I thought the service wasn’t started, so I tried starting it, but that also failed (error text: Failed to retrieve the fizzbaz from the bogatorium)

On first glance, it looks like there’s a real problem here. I try to avoid the “works on my machine” comments, but I thought it was strange that nobody else had seen this. I assumed the Problem was Between the Keyboard And the Chair (PBKAC).

At first, that seemed to be exactly the problem. You see, there is no foo.service. It’s actually called fizz.service, but it’s run as part of the foo toolset, so it’s an easy misunderstanding. If they would have queried for fizz.service, they would have seen it happily running.

Their tests failed because they had an invalid command line for their test (or more specifically, they specified an invalid address for the machine where they wanted to run the tests).

And then they got that strange error message attempting to start the service because they were attempting to restart a service that was already running (interesting, however, was that they did manage to try and start the fizz service rather than the foo service at this point.

Three errors, all different, yet relatively easy to see as symptoms of a common problem. Except they weren’t.

Absolutely, definitely, beyond a shadow of a doubt, PBKAC.

But maybe not. Actually, definitely not. The princess is in another castle – or between a different keyboard and chair.

PBKAC was definitely in play when the user tried to query for the wrong service. My service names above are silly, but our service names are actually nearly as confusing. You only make this mistake once, but it’s not too difficult to make.

When the tests failed, the error said the service wasn’t running. The error was 100% accurate (the service wasn’t running because the user connected to the wrong machine). What if, as a courtesy, that error message said something like, “The service isn’t running. Are you connected to a valid target machine”? I bet that would have set off a few light bulbs rather than generated more confusion.

That message about the fizzabaz and the bogatorium isn’t far from what the user actually saw. What they saw was again, a 100% accurate statement of what went wrong – it just gave no clue of what may have caused it.

Granted, these were internal tools, but that’s a horrible excuse to confuse someone. I bet I’ve seen hundreds (if not thousands) of error messages like this – but very few that offer even the tiniest bit of trouble shooting or diagnostic advice. It’s easy to blame the user when they do something “wrong” (or unexpected), but ultimately, it’s rarely their fault. And in the end, if they can’t do what they want to do with your software, they’ll take their business and money elsewhere.

And then, it’s your problem alone.

Last summer, I posted a short rant on multitasking. If you don’t want to read, it was my normal type of rant where I complain about people taking an already generalized statement and apply it even more widely.

This week, in response to that post (some responses take longer than others), I received a pointer to a little multitasking test that I thought would be fun to share. I’m not affiliated, no kickback, blah blah blah, just sharing because I think it’s interesting. I embedded the test below, but you can go straight to the source at http://open-site.org/blog/the-multitask-test/ as well. (note – the embedded version doesn’t always work that well, so head on over to the open-site.org page if you’re having problems).

Like most Microsoft employees, I have a whiteboard in my office, and mine (also like most) gets used a lot for notes, explanations, architecture, or whatever.

This is nothing new – it’s part of the software culture. A few years ago, I recorded a few handfuls of these talks (some slightly staged) with some all-star colleagues and we posted them on the (now-defunct) Microsoft tester center site. Given that the talks are getting pretty difficult to find, I thought I’d take a few of them and re-post to youtube. I have no schedule in mind for posting, but I expect I’ll re-post 10-20 or so in the coming months.

Here are two to get things started. The first is me talking with James Rodriguez about code reviews, and the second is Alan Myrvold getting me started with Security Testing.