Exploring Testing and Programming

For the last day or so, I’ve been thinking a lot about programming and testing, about collaboration and walls, and about where this may all be going. This post is the start of my mental exploration of the subject.

In the beginning…

In How We Test Software at Microsoft, we told the story of Microsoft hiring our first tester, Lloyd Frink. Lloyd was a high school intern, and his job was to “test” the Basic compiler by running some programs through the compiler and make sure they worked. That was 1979, but full time testers didn’t show up at Microsoft until 1983. I have no recollection or knowledge of what was happening in the rest of the software industry at that time, but I imagine it was probably the same. Since then, Microsoft has hired thousands of testers (we currently employ over 9000), and the industry has hired a kazillion (yes, a made up number, because I really have no way of guessing) software testers (regardless of title).

Now, in the last several years, many teams (including some inside of Microsoft), have moved to the “whole team approach” – where separation of testing and programming roles are blurred or gone (the activities will always exist). Even on teams where separate roles still exist, there is still a significant amount of blurring between which activities each role does as part of their job (for example, I, like many of my test colleagues, make changes to product code, just as many programmers on the team create and edit test code).

The Wall

The metaphorical “wall” between programming and testing is often quite high (as in programmers write code, then throw it over the wall to testers; who, in turn, throw the bugs back over the wall to developers, who make fixes and start the wall-chucking process over again). Lloyd was hired to catch stuff thrown to him over the wall, and I think it’s a fair argument to say it would have been more efficient for developers to run those tests as part of a check-in suite (assuming, of course, they had such a thing).

To be completely transparent, a wall on my team still exists – it’s just a much shorter wall. There are still expectations of programmers and testers that fit into the typical roles, but collaboration and code throwing don’t occur often.

Infection

Elisabeth Hendrickson speaks often of being “test-infected” to describe programmers who “get” testing (I believe the term was coined by Erich Gamma). I for one, can’t think of a good reason for a programmer to not have some test ideas in their head when they write code.

Similarly, I think that testers can benefit from being a bit “code-infected”. Before the self-preserving screams of oppression begin, let me note that I said testers “can benefit” from some code knowledge. I’m not talking about language expertise – a basic understanding of programming logic and some ideas on how computer programs may improve testing is usually plenty. It’s not necessary to know anything about programming to be a tester – I’m just saying it can help.

Beyond the disclaimer, it’s worth elaborating on the last two paragraphs here. I have had testers tell me that knowledge of code will adversely affect the “tester mindset” – that by knowing how to code, the tester will somehow utilize less of their critical thinking or problem solving skills. While I suspect this to be a false assumption, I have no proof of whether this is true or not. But I will hypothesize that if this is true (that code-infection adversely affects testing skill), then it’s just as likely that test-infection adversely effects programming ability. When I look at it this way, the answer clears up a bit. Take a moment to think about the best programmer you know. I’m fortunate enough to work with some really incredible (and some famous) programmers. Every single one of them is test-infected. Yes – I have no hard data, and zero empirical research, so I could be high as a kite on this, but I’d be curious to hear if anyone knows a great programmer who doesn’t have the ability to think at least a little bit like a tester.

Code Infected

So – let’s suppose for a few minutes that code-infection is beneficial for testers. Not even necessarily equally beneficial – just helpful. Is there such a thing as too infected? An old colleague of mine, John, once said,

“I’m not the best tester on the team, and I’m not the best programmer. But, of the programmers on the team, I’m the best tester, and of the testers, I’m the best programmer”.

Which role should John be in? On your team, would he be a programmer, or a tester? Is it a clear cut decision, does it depend, or does it even matter? Is John a test-infected developer, or a code-infected tester? Does it matter?

I think the answer to the above questions are either “It depends”, or “It doesn’t matter”, so let me ask a different question. Do you want a team full of John’s? Maybe you do, or maybe you don’t (or maybe it depends), so let me rephrase yet again in the form of a quick thought experiment.

An experiment

You are forming a nine-person team to make a super cool software application. You get to build a dream team to put it together. On a scale from 1-9, where a 1 is a tester with zero knowledge of programming, a 9 is a developer who can’t even pronounce “unit test”, and a programmer / tester like John is a 5 – which employees do you hire (note that all of these people get paid the same, work the same hours, get along with each other, etc.).

clip_image002 

To be fair, I don’t think there’s a “right” answer to this question, but we can certainly explore some options of what would be more or less “right”. One option is to (conveniently) select one of each type. Then you get infection-diversity! However, I’m betting that most of you are thinking, “There’s no room on my team for developers who don’t write unit tests!” So, is “some” testing knowledge enough? Is it possible to have too much? If I hire 7’s and 8’s, do I eventually want them to have more testing knowledge? If that’s the case, should I just hire 5’s and 6s instead?

And what about testers? Remember, that code-infected doesn’t necessarily mean that the testers are code-monkeys cranking out tools and automation all day long – it just means that they have (varying levels of) programming skills as part of their available toolbox. The context of the product may have some influence on my choice, but if you agree that some programming knowledge can’t hurt, then maybe a 2 is better than a 1. But is a 3 better than a 2? “I suppose that “better” depends, but to me, it’s a worthwhile thought process.

But that’s not right…

If you’ve read this far without jumping down to comments to yell at me, let me admit now that I’ve led you in a bad direction. The exercise above sort of assumes that as you gain testing ability, you lose programming ability (and vice-versa). That’s not true in the real world, so let’s try a bit of a twist and see if we can be slightly more realistic. For this exercise, you have the $1,000,000 of salary to spend on a maximum of 10 employees. You have the following additional requirements:

  • An employee can have a “programmer” rating of 0-10, where 0 is can’t even write a batch file to 10 where they dream in algorithms.
  • Similarly, an employee has a “tester” rating of 0-10 where 0 is no knowledge of testing, and is at least the best tester you’ve ever dreamt of.
  • Minimum salary is $50k, and the maximum salary is (arbitrarily) $160,000.

Because context is important, let’s say the app is a social networking site with clients for phones and tablets, as well as a web interface. Your exercise (if you want to play) is to think about the type of people you would use to staff this team. One option, for example, is to have five developers, each with a 10 rating (5x$100,000), paired with 10 testers who also each have a 10 rating (5x$100,000). Another option would be to have 6 developers, each with a 15 rating (e.g. 9-dev and 6-test for  6x$150,000=$800,000) paired with 4 testers, each with a 5 rating (4x$50,000-$20,000). The choice is yours, and there’s no right answer – but I think the thought exercise is interesting.

I’ll close this post by saying that I really, really hate sequels and series when it comes to blog posts (and I’ll add an extra “really” to that statement when referring to my own posts). But there’s more to come here – both in my thoughts on testing and programming, and in more thought exercises that I hope you will consider.

More to come.

Comments

  1. I like your thought problem, but from personal experience I can say being a “John” is perhaps the worst place to inhabit if you are attempting to find a place at a company or on a team. You aren’t a strong enough programmer to inhabit the role of a traditional “Development Engineer in Test” and you’re too technical (ie: expensive) to inhabit the role of a “Software Tester.” Agile shops tend to be more excited about finding a skilled generalist who can inhabit multiple roles, so perhaps that’s changing as actual agile has more adoptions.

  2. The reality is you never have a team of testers/engineers of similar skill sets much less multiple plp with cross pollinated skill sets. If you take that; with the reality that you rarely have the time required to create unit test and/or UI automation framework as you’re developing a new apps (things change too quickly). Projects rarely run smooth and you frequently have to change and iterate as you go; making any early attempts at creating automation wasted.

    Admittedly in the academic since things wouldn’t be this way and projects would always run on time with no mistakes and you would never have executive/customer feature creep. With all this said I would take these resources.

    I am making an assumption that a stronger/better engineers would level 8-9 and stronger/better testers would be level 1-2 (sorry if this is wrong).

    I would want 3-4 testers with 6-7 engineers.
    Testers (minimum 3)
    1-2ct. level 1 tester: Basic testers that dive through the manaughtness/iterative tasks of feature, integration, performance, etc… testing.
    1-2ct. level 2 tester: The goal would be that these people understand the SDLC so that they understand how feature development (code changes) changes have rippling effects throughout the application under test. There code understanding is not such that they want to program, but provide a knowledge is power mentality (test professionals as opposed to wanna-be developers) to compliment their testing prowess.

    6-7 ct level 8-9 engineers (split): This is a good mix of straight engineers to plow through feature development and engineers with testing knowledge that help them to write solid error resilient code.

    hope you enjoy

  3. Hello Alan.

    So I think what you are describing is an SET. A software engineer in test. Somebody who knows code can contribute to the coding done by the devs and can help write testable software.

    I also wrote a post about that that it’s always beneficial to know the subject under test. Not only in software world but in testing world in general.

    Here is a link to the post if you care to read it.

    http://ramblingsofaswtester.com/?p=116

    Thanks.
    Gergely

  4. +1 for testers with T-shaped skills. I love to do a little coding – automating tests was always fun. But a couple of years ago I realized I’d never code automated tests as well as the programmers on my team who write code 8 hours a day. It’s much better and more fun to pair with them to do it.

    Technical awareness helps testers and programmers communicate. So does “testing awareness” on the part of the programmers. I’ve been living this dream most of my career, but especially since 2000 when I joined my first “agile” team.

  5. I’ve been testing apps and systems since 1994.

    The best programmers I’ve seen are the ones who think about testing, not just what they’re building.

    Whereas I’ve also worked with programmers who do not think beyond the spec.

    Guess whose code is easier to test?

    That’s right, the programmers who do not think beyond the spec! Why is that? Because I know all their mistakes. The patterns are easy to spot. They’re predictable.

    Whereas, the programmers who think about testing, well, they’re difficult to trip up! I have to think more intensely. They don’t hand me defects on a platter. It’s harder work, and it makes me a better tester in the long run.

    But why limit a software tester to having some coding experience? What about having a software tester on your team who used to be an art major, or an English major, or somebody with a Master’s Degree in Psychology? Does that not add diversity?

    And why do so many programmers who write about testing believe that “coding” is the only additional skill set that testers need? I see this sort of column or blog post all the time.

    1. > why do so many programmers who write about testing believe that “coding” is the only additional skill set that testers need?

      Probably because it’s so prevalent in our industry to say “testers can’t code” (“can’t” as in aren’t able to, or “can’t” as in should be kept from doing so – either way), and for so many testers to believe it.

    2. Easier to test? Or easier to find bugs? To me, those are two different things, and I’d expect that those are drastically different between those two groups.

      To me, easy-to-find bugs are often a sign of a team (or project) out of control.

  6. Awesome post…

    I always found the argument that learning code will make you a worse tester to just be insane, and I’m glad you’ve addressed it a bit here. If anything being able to code has helped my testing skill, my communication with the team, respect from development, and general awesomeness.

    I’ve being thinking about your team challenge for a while and I’ve probably put in more thought than i should have so here goes.

    So we have a social app so i’d want a fairly high dev to tester ratio. Since we aren’t launching a space ship or running a bank I think the risk out weighs the benefit (but my top level testers could write code too! I’m going to table that one at the moment). In my experience I’ve always found that a great group of people working as a team is the most important thing, and I’d want people at the top of the industry.

    So I started doing the math… ugh. I could only afford 6 devs at 150 and that leaves not much for testers. If we reduce it to 135k we could have 6 devs and 2 testers at 95k.

    I’d try to get 2 interns (FREE) on the team as well. I feel that they offer, energy, ideas, and super cheap work 🙂

    Then I thought, even if I had the funds, could I even create a dream team? Probably not, it’s really, really, really hard to find people (at least from the interviews I’ve done) that fit the type of profile I’d want to work with.

    So that’s my answer at the moment… I’m still not sure I’m happy with it. At the end of the day I’m really glad I don’t make these large decisions, I never really thought much about building a team from scratch.

    1. Love the intern idea :}

      I wonder if anyone has explored the idea of what a team would look like – or what they could do if the testers were at the high end of the pay-scale. Testers (or test-minded people) with the impact to earn that money typically can influence a programming team to make better quality decisions.

      Imagine, for example, a project with 5 programmers at 125k each working with a tester (you’d probably have to call him something else) at 200k?

      Seems to me like it could work.

  7. Not too long ago I found myself in a calibration meeting with a GM whom I (still) respect doing the “desert island” / “life raft” exercise, and there was no way a tester was getting into the top slots b/c after all if you only had a tester, you have no product, right? 🙂

Leave a Reply to Mark Rushton Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.