Death and Testing

I’m heading off for a long vacation today, so this is likely my last post of the year. It’s been a crazy year, and I thought I’d end it with something that a lot of my recent posts have been leading to (e.g. this post on tearing down the walls).

Some of you will hate this. I’m not sure if it’s because you’re afraid of your job, or because it’s so far out of your world that it doesn’t make sense. For you, let me tell you about the audience who will read this and say, “ho-hum – tell me something I don’t already know”.

Testing as we know it, is changing – and the changes are big (for some folks, at least). Testing as an activity will always exist. But I see test as a role going away – and it can’t disappear fast enough.

Test Is Dead (or is it)?

A few years ago, my friend James Whittaker angered testers world-wide with his talks on Test is Dead. James premise was that for a whole lot of software out there, it’s more cost-efficient to get data and issues from customers actually using the product than from a test team pounding on the product. Unfortunately, way too many testers failed to see the context and big picture of the main point and screamed loud and wide about how quality would suffer without testing.

But James was far more correct than people give him credit for.

Hiring testers to pound quality into a product after it’s been developed is a waste of money. The more I think about it, the more I realize that Test Teams are an artifact of older, predictive (staged / waterfall-ish) software development models. The notion that one team (or part of a team) is responsible for writing code, and another part of the team is responsible for making sure it works as expected seems wasteful to me (even after spending the majority of my career doing just this).

Test teams are dying. You may disagree, but I encourage you to take a critical look at whether you believe a non-integrated test team is the most cost efficient way to evaluate a product. Yes, I know there are companies formed entirely around evaluating already-developed products – but I believe that the days of testing-quality-in (the equivalent of the people who clean up after animals in a parade) are ending (or ending, at least, for the quality software products of the future).

Testing is NOT Dead

Before you skewer me with your protests (and some of you will, despite my best efforts to explain), let me reiterate. Testing is not going away. The testing activity will thrive and improve and evolve. It’s just time to think about getting rid of the old ways of thinking about who does the testing and where it happens.

Plain and simple, we (the software industry) have been wasting money for years letting testers play safety-net for lazy developers. It’s much more cost effective and efficient for developers to write unit tests and some or all functional level tests – and maybe more. Other developers on the team (many of them former testers) should focus entirely on “ilities” such as reliability, performance, security, etc. as well as evaluating end-to-end scenarios (epics) spanning multiple features. (related paper from me here – it’s a few years old, but I think it foreshadowed this change nicely).

There are other activities – like data collection and analysis (feel free to insert the buzzword “Data Science” here if you wish) and other tasks we need software teams to do – but we don’t need separate teams for those tasks. We need people to do those things, but those specialists can work anywhere. Why is it that I’ve seen just as many performance or reliability teams reporting to development managers as I have seen reporting to test managers? Some of my colleagues talk about the need for testers to “transition” to data science or other roles. In reality, it may be a transition, or it may be different people. It doesn’t matter (what remains true, is that we will have an increasing need for people who can analyze data). I think one of the only reasons we don’t talk about having a specific “Data Science Team” is that we already have enough disciplines in software engineering, so we default to shoving them into the best suited discipline for analysis roles.

Whole Team Approach?

We need integrated teams – not separate teams. We need teams of generalizing specialists to make great software. The disciplines and teams get in the way of efficient creation of quality software. Some say that the programmer/test relationship provides checks and balances – but I see no reason why those discussions and positive conflicts can’t happen on a team where everyone works tightly together.

I’ve seen a few mentions of the “whole team approach” to software creation, but that term (to me) doesn’t adequately describe how great teams (can) work. In a team full of generalists, everybody can do everything, but nobody does anything particularly well. That doesn’t work if you’re trying to make quality software. In a team of generalizing specialists, everyone does something (or a few things) really well – but the team members are adaptable when necessary. Teams need specialists in design, user interface, performance, reliability, deployment, internal infrastructure, data analysis, and scads of other areas. Teams also need people who specialize in testing – specialties that include end-to-end evaluation and risk analysis, as well as coaching developers writing unit and functional tests.

But the separate team for test is dead. It’s a waste, and we need to stop thinking about putting our specialists on separate teams.

Testing is a Specialty

I’m looking at a twitter feed full of comments from testers worrying about what this means to their careers. A good tester, as part of a team of generalizing specialists is worth every bit of salary and value as any other specialist on the team. Test “specialists” who can’t provide this level of value, however, probably should worry about their career. Test may be dead, but good testing will live and be valued for a long, long time. But we need to stop thinking that we need test teams to pound quality into a product, and figure out how to fully integrate great test minds into great software teams.

There’s a subtlety here that goes beyond semantics. Jobs (most of them, at least) aren’t at stake here. The great testers of today will be great test specialists on software teams tomorrow. It doesn’t’ matter if they code or not, as long as they provide value and contribute. I’ll repeat that, because I can almost guarantee a comment about how stupid I am for proposing a world where testers need to code. Test specialists won’t (necessarily) need to code. Many of the “test automators” today will write features, as well as functional tests (using their mad automation skillz) – and they’ll coach the programmers on how to do the same – leveraging the specialties of each other.

A glance through any testing community will show dozens of definitions of what testing is, and what testers do. It’s too much. It doesn’t matter. We need to stop building organizations where one team writes code and another team tests it. Testing needs to happen all of the time, and at the same time as development. For this to happen, we need to get testers out of their test teams, and fully integrated into software teams.

A Better Future

Today, testing articles, posts, and tweets are filled with whining (or mere complaints) about the sorry plight of testing, the lack of testing respect, the difficult of measuring testing, and other worries of the testing community I’ve been hearing for the past 20 years.

But a new world where testing is merely a specialty within the software team eliminates these worries. There’s no more us versus them. We have a team, and we each supply different specialties that contribute to the team’s success.

Preemptive post script

I’m leaving (on a jet plane), and I half-rushed this post out. I half (or full) rush most of my posts, but this one is important. I’ll try to respond to comments (and hate mail) when I land, but I expect I’ll have follow ups and re-writes coming. I’m positive there will be much speculation on what I didn’t mention, so I’ll deal with that soon as well.

More will come.


  1. I agree in principle but my agreement is fairly moot since this is the way the industry is going whether I like it or not. However I will say having an independent test organization is a very good idea from a personnel or organizational chart perspective. Nearly all software projects will benefit from having specialists like the ones you mention. However just like you wouldn’t want to have an oncologist advise a thoracic surgeon, it’s generally helpful if you have test specialists mentor, advise, and manage other test specialists.

  2. I do only testing, and our developers wouldn’t know a test if it hit them in the head, so you would think I would be one of the first to get defensive about this. I’m not, and here’s why:
    I sit with the developers, report to the same manager, and push for tighter integration and better, faster feedback between us every chance I get. The results of this have been noticeable in our improved quality over other products in our company, who either fall into the category of an agile team with no testers, or more commonly a separate, sometimes overseas only, test team.
    I have little hope that we, or many companies, will reach your ultimate goal any time in the near future. But I do believe any movement towards tighter integration between testers and developers is a good one.

  3. I love the analogy about the parade animals! The concept of the generalizing specialist is very core to the whole Agile mindset and having a whole team that is dedicated to the quality of a piece of software rather than trying to bake it in at the end makes a huge difference in the quality of the end product. Excellent post!

  4. I remember reading How Google Tests Software and not really agreeing with James Whittaker. I felt like he was saying that testing is dead, therefore we don’t need testers anymore. This explanation makes much more sense to me.

    I really love the idea of moving from “test last” to “test always” or “test continuously”. All stages of development from the first idea to the final build before release can benefit from testing, whether it’s from a developer, product manager, dedicated tester or otherwise. If this means getting rid of QA departments so be it. I hope the ideas in this post end up going far and wide in the software world.

  5. You should probably address a few things that you didn’t in your article. I have been in T&E all of my adult life and testing has always been squeezed until the tail end of a development effort that has a finite deadline. Testers don’t often get much respect for their contributions. More often than not they become an unnecessary after thought. Same is true for our IA counterparts unfortunately. The byproducts of testing are often reused for future test events. For example, disciplined test procedures are reused and leveraged for the next or upcoming test event. Trainers leverage test procedures to develop training curriculum, IV&V teams often times request the original test team for the same information. You almost always need a detailed test report with findings, problems, results and recommendations before the client will field the software or system so that he/she can justify their decision. These byproducts of T&E cannot be under estimated. Wish I had more time and a laptop to fully state my views but sadly I am forced to respond with my droid. 😉
    In principal I get what you’re saying but I still think the old way of doing business still makes sense. There is probably some middle ground but I have also always subscribed to the “If it ain’t broke don’t fix it” mentality. I am open minded about this and hope to read more. If I can improve my skill set and do this job better I am ready to listen. Thanks!

  6. I normally don’t believe any sort of generalization statements. However, the software products and the way software are developed is quite different from 10 years ago. New technology, new way of project management like agile, SOA development, web application, game console, smart TV, medical device, mobile and etc. Pretty much everything in our life consists of software. And different companies tries different way to produce software. Start up companies cannot operate as similar as a quite mature company. Just like Alberto mentioned in his keynote in GTAC 2011. Idea bugs will be far more serious than functional bug for start ups. But for company like Amazon cannot tolerate 10 mins of downtime due to minor software bugs. How Azure cloud DB team testing would not work for iPhone app testing. What Boeing software team will need from their testing would be different from what Google+ team need from their testing.

    We know importance of human mind in software testing and we also understand the benefit of automation. And we’ve been building good knowledge and strategy around testing. What’s important is to understand the development process and situation around the testing work and provide most appropriate testing solution.

    What is obvious is that if you only know manual testing, you would not be valuable for software companies/teams that need automation. If you only believe in agile, you will not be valuable in a company that have to do non-agile way. If you don’t believe the value of continuous integration and continuous deployment, you’ll not be valuable in a company which believes in it. Different companies and teams operate differently and there are so many variables in software development. Have a wisdom of human mind in software testing. Have expert execution in coding, code review, designing and developing automation. Have a BIG tool set. Then when you come across a testing situation, use your experience, knowledge and expertise to provide most appropriate testing solution.

  7. Alan,
    Crap man! Guess I have been using my ZE degree all along (cleaning up the elephant dung at the zoo) in my testing work.

    Provocative is the best way to describe your post. But you are right in the premise that Testing needs to be more tightly integrated into the ‘team’/process from day one. Which I think is happening, and has been for a while now. The whole idea of ‘team’ is important and the focus of certain specialists is needed. That is why the Team Model that both RUP & MSF emphasize needs to be universally accepted and used. Agile practices/methodologies (IMO) do push for this, but not enough because it is a developer centric approach (again, IMO) and need to really strive to use the ‘generalist specialist’ you talk about.

    But also too with agile we have seen a bit of return to the cowboy way of development by developers (and some companies) and that has caused some major headaches. The main principle I see you pushing for is that of ‘all of us’ are involved in testing and thus quality on our products. Everyone on the team needs to think about testing and baking quality (as it is perceived) to the product. We all have an equal stake in the product.

    Now should the testing group/department disappear. That I can debate… I think it depends on the situation and maturity of the overall company, development team and processes. There does need to be some oversight/guidance, and how that is implimented can vary. BUT you do not want a situation (like way back when in the late 80’s/early 90’s) where development ruled over everyone. That is the Fox watching the Hen house, and I really prefer not to go back to that situation.

    So there is a way forward and you propose/highlight one of them. I guess the only thing left to say is “let’s see what happens next”. Hopefully 2014 will be a very interesting one for everyone.



  8. Alan,

    I applaud you for bringing up this interesting topic and risking the ire of your fellow testers.

    W. Edwards Deming facetiously said: “Let’s make toast the American way. You burn. I’ll scrape.” Taking a systems thinking approach, Deming made those comments decades ago while advocating for approaches that would generate fewer problems in creation processes themselves as opposed to spending time and effort after those processes to identify and fix the problems. His message was that sensible well-run teams should put more attention into improving our processes to eliminate the need for so much toast scraping (elephant cleanup).

    Were Deming’s comments made with (primarily) manufacturing contexts in mind? Yes.

    Can people get into trouble over-generalizing about applying lessons from manufacturing in software development and testing? Yes.

    Even so, can teams find ways to more fully-integrate testing activities into the software development process itself and, by doing so, reduce the relative amount of effort required for after-the-fact “burnt toast testing”? As you suggest in your post, I also think the answer here is yes. For all of our sakes, I sure hope so.

    – Justin

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.