Taking a break

I’m taking a vacation. usually people say a “much needed vacation”, or a “long overdue vacation”, but in my case it’s just a vacation. My new job is keeping me interested, engaged and I’m having fun – truthfully, I can’t wait to get back, but it will be nice to spend some time with the family for a few weeks.

I do think that I need a break from the blogosphere and twittterville. I’m immensely excited about the number of new test blogs and test tweeters appearing these days, but I’m sort of bothered by the lack of advancement in testing ideas – it seems that people just keep on talking about the same things over and over, and keep on rehashing old arguments or statements rather than exploring new test ideas. I get it – if you do stuff wrong, it doesn’t work. There are no silver bullets. Testing is hard. Don’t insult my intelligence by telling me those things again. Even if you use all caps or new metaphors, it’s the same statements again and again and I’m getting bored.

I’ve (almost) always taken a high road with my community participation. I sometimes see popular testers tell half-truths or exaggerate their accomplishments in their diatribes to the masses, but it doesn’t seem right for me to call them out and set the record straight (even if I frequently wish someone else would). I’m not about to make every post a rant, but may not bother to try to appeal to a wide part of the testing population either. I guess the point is that I’m not seeing what I want out of the community or my own blogging, so I’m hoping that by not reading or writing blogs or tweeting for the next few weeks that I’ll figure out what I want. If not, I’ll just take more time off until I figure it out.

Feel free to leave comments – if you haven’t commented on this blog before, comments may sit in the moderation queue for a while. The rest will go through unedited until I get around to playing with the internet again.

Edit – 3:00pm, July 19

It doesn’t matter who I’m referring to above – if you think I’m talking about you, I probably am not talking about you – unless you’re sure I’m talking about you, then you’re only probably wrong.

Also – I’d like to point out that I’m a huge fan of all testers actively engaging and learning about the craft. Groups like weekend testers and individuals who explore new approaches are high on my “love” list. Also note that many who think they are actively learning and engaging actually aren’t doing either. If this statement bothers you, please refer to the above paragraph.

Thinking about bugs

I probably haven’t mentioned in a while how nice it is to be back on a product team. The complexity and dynamic nature of software development is something I missed more than I knew, and I’m having fun being back in the flow of shipping a product for a million or so users. One thing that I never got a chance to think about while I was in EE was bugs. To be clear, I got to think about the concept of bugs, but not about real live bugs. Bugs are a big byproduct of the testing process, and like it or not, they help dictate some part of the flow of software development. At the very least, they give us something to think about.

So let me share what I’ve been thinking about…

Every bug is more difficult to find than the previously found bug. There will be little hops off the axis, but in general, I believe this to be true. Imagine if the opposite were true – if every bug were easier to find than the previous bug, we’d find more bugs every day and eventually have a product consisting entirely of bugs (insert joke about your least favorite product here).

As bug discovery decreases, product quality improves. There are plenty of reasons where this may not be true, but if few bugs are being discovered, it probably means that product quality is improving (or that all of the testers went on vacation) – but …

Tester skill and knowledge increases with each found bug. When a tester finds a bug, they also acquire knowledge. They learn something about the product, or a technique or behavior that they can use to find bugs in the future.

Tester skill can’t grow as fast as bug discovery difficulty decreases. If tester skill improved faster than the difficulty of finding the next bug increased, we’d find more bugs over time until we had a product with more bugs than …yes – you see where I’m going.

<some made up line charts would do well here, but I’m feeling lazy>

What I’m struggling with is coming up with a better understanding of how these statements interact. In more concrete terms, if the number of bugs discovered in a product or feature area isn’t tapering off at “an expected rate”, does that mean that product quality isn’t improving as much as expected, or that tester skill is improving more than expected? Since the answer is (you guessed it) “it depends”, how do you dig deeper? How do you know the right answer with some degree of confidence?

I have answers for this that satisfy me (customer data is a big part of this), as well correlating a bunch of other measurements, but I wonder if anyone else thinks about stuff like this and has other thoughts or ideas or experiences to share.

More on careers

I left out a couple of obvious things in my career post yesterday and thought I’d write them down before I forgot.


Yesterday, I wrote about the non-management career path for testers at MS. There is, indeed, a career path for managers – I just tend to talk about the non-management career path because that’s where I’ve tried to keep myself. We have two titles for people managers at Microsoft. We call managers who manage a team of non-managers Leads, and call managers who manage leads Managers. In other words, a Test Lead manages testers, and a Test Manager manages test leads. Got it?

Generally, Test Leads are in the Senior band, although it’s somewhat common to have leads in the upper ranks of the SDET IIs. The level of knowledge (and more importantly, amount of different testing experiences) testers have by the time they get to these levels gives them the confidence, ability, and credibility to be successful. Leads are expected to get some “real” work done in addition to managing their team. Test Managers are typically Principal or higher (although, as with Test Leads, may be in the upper bounds of Senior).

Something else to note is that the Lead role does not necessarily need to grow to the Manager role. If you are a lead, but never want to manage a large team of leads, you can grow to Principal+ levels as a lead. Or, you can be a lead for a while, bounce back to being an individual contributor, and bounce back to being a lead if you desire (and if the opportunities and business needs are there).

Another take on career stages

I recently re-read Spolsky’s Smart and Gets Things Done. That (the title, and the description from the book) is pretty much what SDETs and most of the SDET2s do. Senior SDETs (and SDET2s who are on their way) move to a stage of Smart, and Makes Stuff Happen. It’s one thing to know how to do stuff really well, but leaders are able to make stuff happen through influence (and the help and support of others). We expect Principal’s (and those about to be Principal) to think more about How Stuff Happens. Strategy, decision making and and solid systems thinking all feed into the “how”. Of course, you still need to dive into making stuff happen and getting things done once in a while, but setting the right direction, for the right reasons, is critical.

Careers in Test

I’d like to elaborate on something from a previous post on SDETs at Microsoft. One thing that is perhaps a bit different about the test role at Microsoft is its tie to the career path. We are, for better or worse, big believers in career growth and expect growth in scope and impact from all of our testers.

Eric, a former manager of mine has a nice post on our “career stages” in this blog post. His post is on SDEs, but it applies to SDETs for the most part. Eric’s descriptions are wonderful, and well written. At risk of being a copy cat blogger, however, I’m going to offer a sentence or two of commentary on the non-manager SDET career stages at Microsoft.

SDET – This is where most of our testers start, and where you figure out how to do your job, and do it well. This stage is about learning the trade (tools, practices, dynamics, etc.). You won’t make it out of this stage just by being a fantastic tester – everyone in this stage is a fantastic tester.

SDET II – At this stage, the big thing is independence – SDET “2s” get stuff done, and don’t get stuck – in fact, they don’t let themselves get stuck. At this stage, testers also start to get their eyes of the immediate tasks at hand more often and think a bit about the future of their org. They have influence in their team, and often accomplish significant tasks that impact their entire org.

Senior SDET – Leadership is expected at this stage. It can be technical leadership or strategic influence – but the big point is that you have a huge amount of influence and impact on your entire team (with occasional forays into impact across your organization). For a longer explanation, please meet Alecha, Sanjay, Jodi and Kaz.

Principal SDET – SDETs at this level are expected to have influence and impact across their group (i.e. well beyond the testers that work in their own managers organization). We don’t have personas for this level shared externally yet, but if you think of the gang of four above with a much wider scope of impact, you’ll be close.

Partner SDET – This is scope of impact ratcheted up to 11. These folks leaders across an entire division (remember, that we’re talking about leadership – not management). They often make decisions that can save millions of dollars (and do so with consistency and confidence). Again – we have no personas for these folks, but just think of the same gang of four leading successfully across an entire division.

Internally, we have career stage profiles that define a lot more about these roles (some examples are in chapter 2 of hwtsam). The one bit of possible weirdness with the big emphasis on career paths is that we expect everyone to grow to at least Senior, and I worry sometimes that there are fantastically awesome wonderful super-testers who just aren’t capable of leadership at the Senior level. As I think about it though, I wonder if I (we?) just need a broader view of what leadership in testing really is. Right now, we have a relatively small percentage of folks at Senior+ levels, so I may be looking at this through eyes stuck on the small population of people currently “leading” at those levels.

Just hoping to clear up some mysteries – hope you find this helpful.

Some random stuff

I have some random stuff to share – most of it not worthy of a post, so I’m consolidating.

  • I’ll be heading to Toulouse in a few weeks for a bit of a vacation. Looking forward to the time off, as well as food, wine, and warm weather. At one point in my life, I was almost adequate in speaking a very broken form of French (one that only included present tense verbs and lot’s of food terms) – but I’m pretty sure I’ve forgotten even that little bit of the language these days. Hopefully enough will come back before I embarrass myself too much.
  • My “new” job (I’ve been here 4 months already) is going great. I’ve had some pretty good success with exploratory testing on the team lately, and it’s an effort that a lot of people are seeing value with. I’m also working on some stuff with code coverage, test code quality and code reviews that are keeping me busy.
  • Buried in this post from just over 4 months ago was a comment about my rising blood pressure. I’m happy to say that as of this morning (after a visit to my doc), my bp has returned to it’s historic norms (actually, a bit better – 110/66). Better yet, I mentally and physically feel better these days (and this is before my vacation).

I have a follow up on my SDET Pendulum post drafted, and will try to get it up by the end of the weekend (for those of you who like my longer posts).


I’d like to announce my candidacy for…no – not that kind of politics, I thought I’d drop a few comments on office politics.

Too often, people see “politics” as the evil underbelly of the corporate world, and that competency in office politics requires that you can backstab your coworkers with no remorse, find a variety of ways to undermine your colleagues, and take credit for the work of anyone who won’t fight your claims. This definition from dictionary.com covers it well:

to deal with people in an opportunistic, manipulative, or devious way, as for job advancement.

But to me, that’s a cynical, shallow, and completely unhelpful view of what politics really are in the workplace. However, I do like the first four words – “To Deal With People” – in my happy rose colored view, that’s the part of the politic game that everyone has to play. I think that long term success in any workplace requires that you can deal with people. If you want influence, you’re going to need to have allies. If you attempt a difficult change, you may also annoy or disappoint some people. That’s all fine, as long as you find a way to talk, listen, or do whatever it takes to keep the wheels moving. I’m not saying that you have to kiss everyone’s ass, but if you want to be a leader – someone who can influence people to change, improve, or flat out take a chance on something that you want to do, you’re going to need to find a way to deal with them.

There may be people in the world that you don’t like or respect. That’s ok –we all know those people. Those of you who are politically savvy know that you probably still need to be able to communicate with them occasionally, so you don’t burn bridges. Who knows – you may need those people as an ally someday, and if you flipped the bit on them (and told them that you flipped the bit as well), all I can say is good luck getting that next “big idea” to fly.

More importantly, you need to be able to influence the people who can help you. Chances are good that you have more great ideas than you can take care of yourself. That’s ok, because great leaders take pride in making those around them better, and your great ideas are probably the thing that will make all of your colleagues better.

As long as they’ll listen to you.

And they will listen to you, as long as you are worth listening to. So build credibility, help them in their own projects, give them credit, and raise them up any way you can. In other words, use politics (the good kind) to get your way…and become a leader.

The SDET Pendulum

At a recent internal forum, I hosted a panel discussion comprised of “senior” level testers at Microsoft. The panel was evenly split between managers and non-managers and we took random questions from the audience on career paths in test. The testers in the panel had experience ranging from 11 to 24 years in test. Some of the questions strayed slightly from the topic (e.g. “What does the future of test look like to you?”), but for the most part went well. The session was one hour, but could have held it’s own for at least another 30 minutes.

That’s pretty much it, so I suppose I can end this post here…

But one question came up that I want to dig into a bit, and I know that some of you who occasionally read this blog have some opinions on the subject, so fire them away please.

The question in mind may as well have been seeded by me, because it’s a subject I’ve brought up before. The question was: “Has Microsoft swung the pendulum too far in hiring only programmers as testers?” I can’t remember if my answer was “No, but Yes”, or “Yes, but No”, or “It depends”, but I’ll give a (much) longer version here for your benefit.

BTW – I would bet that if I asked most non-microsoftees the response they would give for this question, they’d say, “of course – my {cousin | dad | sister | friend } who is the best tester in the {company | city | country | world } can’t get a job there, and they’re {fantastic | awesome | superman }. They just can’t write code.”

And, in many cases, they may be right – but maybe not – or it depends. Some context is in order. I’m not the company expert on this, but my opinion and view of history is as good as any.

Our software is big and complex. We deliver platforms, operating systems, server products and put them into huge deployments. We need testers who can program to help solve the big testing problems that our systems present. Say, for example, you’re a tester on the Exchange team, and you need to verify that messages of varying lengths travel from one endpoint to another successfully. No problem – send mail to a test account and make sure it gets there… well…that solves one happy path. The reality is that the tester needs to send multiple mails through the system with a variety of headers and sizes to ensure that anytime the message is broken up that it still ends up at point B correctly. And, going through outlook is probably not the most efficient way to test this, so it’s probably better to construct the messages programmatically then send them through the system. Oh yeah – there are an almost infinite number of ways the server topology can be constructed, and those settings make a difference too (in other words, we need to deploy any arbitrary topology automatically on demand for these tests). As you can see, there are some obvious benefits to having someone who can program as a tester.

Microsoft has always had testers that can write code. Always. For much of our history, we’ve also employed testers who don’t write code (but who are still good testers). We’ve found over the years that our best testers and best test leaders had a programming or CS background. I’m guessing we also found that defining the career path (MS is really, really big on career paths) for testers was a bit easer to map out when we took programming background and application into account. We still use a lot of non-programmer testers in contract and vendor roles, and I’m guessing we’ll continue to do so. There are many great testers who have learned how to program, but over time we found (or I should say, my personal experience was) that when we fished in the pool of programmers for good testers, we found a much higher percentage of good people than when we fished in the pool of good testers looking for people who could write programs. So, for full-time “blue-badge” employees, we fish for testers where programmers live; we get what we want, and everyone is happy.


Note that in the above I referred to testers who can program. Where I fear we make the mistake in hiring is when we (or anyone else who hires “SDETs”) hire programmers to do testing. The difference is subtle, but important. A tester who can program is a tester first – but they are a tester who can rely on computer knowledge or programming skills to do their work better. A programmer who tests writes tools and automation first – hoping that it will help them be a better tester (hint – it doesn’t work very often). For the most part, Microsoft has testers who can program – and they are freakin’ awesome. The problems they solve and the way they go about it makes me proud, excited, happy, (and a bit jealous) all at once. It’s what SDETs should do.

The industry perception of the SDET appears to be a role of writing automation and tools all day. When I hear this, I think of all the great testers I work with and tell people “No, No – just because our testers can program doesn’t mean that’s all they do all day – they are testers first”. Then, whether it’s in 5 minutes or 5 days (but sometime soon after), I’ll see a blog or a tweet or some other sort of message from someone saying “I’m an SDET – I write automation and tools all day”, and I realize that there’s still some work to do. It doesn’t mean that most of our SDETs write tools and automation all day, but there are certainly pockets. My thoughts are that if you want to write tools and code all day, get out of my business – go be a developer. You’re not helping me (and you’re probably slowing me down), so leave me alone. I want to test, and I want to work along side people who feel the same way.

In the panel discussion I asked every hiring manager in the room to ensure that in the next interview they conducted that they looked for testers who used “technical” skills to solve their problems. Instead of making them write code on the whiteboard, give them hypothetical testing tasks and examine how and when they used the power of the computer to help them. For example, if I ask an interview candidate how he would know if a line drawn on the screen was straight and he started writing up an algorithm to scan the pixels on the screen and determine if the offsets were consistent with a “straight” line, I wouldn’t hire him (“Look at it”, or “use a straight edge while looking at it” are both acceptable answers). However, if I describe the message transport scenario I used a few paragraphs above and their answer is to ask everyone in the company to send a bunch of mail, I probably won’t hire you either. (I’m working on an internal presentation on hiring testers – I’ll come up with better examples :})

Every business is different, and your need for testers or “SDETs” will depend on your business. My call to action is this: If you wan to hire an SDET – hire a tester who can program – not the other way around”. You and your product will be much better off.

It’s a big world (of testing) out there

A few weeks ago, I talked about some presentations I was involved with for internal audiences at Microsoft. Joe Strazzere asked if I could share slides or elaborate, so here goes (on one topic at least).

I’ve never met Joe – but I feel like I know him a little. I interact with testers regularly that I’ve never met. I take time every day to see what testers are doing, what they’re trying, and what they’re thinking. The problem is, that if you’re reading this, you’re probably the same as me. You look beyond the walls of your job to discover new ideas about testing. The talk I helped with (Joshua Williams actually ran the talk) was about looking outside the walls of Microsoft (or wherever you work) to discover what other testers were doing and what they cared about. The best testers I know have a passion for learning – but more importantly, they seek out new ways of learning, rather than just reading one source, or following the work of one test blogger or consultant.

So – I gave them options. I talked about the testing stack exchange, software testing club, test republic, and sqaforums. I talked about Software Test & Performance…I mean Software Test & Quality Assurance magazine and Better Software. We have a license for the IEEE Explore site, and I unveiled the world of academic papers on testing (and above all stressed the importance of critical thinking when attempting to suck down this info). I tell a story sometimes about the first testing book I read. It opened my eyes – after reading it, I felt like I knew everything (I was too naive to challenge it). I read a second book on testing and noticed that it contradicted the first book sometimes. It wasn’t until I read my third (and fourth, fifth, sixth and more) that I began to form my own opinions about testing. Forming your own opinions – rather than just repeating what others say, is something I value tremendously in testers I work with. I also gave a nice plug to weekend testing, as the movement is all about learning and testing.

We also talked about open source. We’re not big adopters of oss at Microsoft, so a lot of people don’t know much about what’s out there. I said what I could without pissing off our lawyers too much.

That was pretty much it. There were some good tangents in the discussion section. Somebody was disappointed that we just didn’t tell them what was happening in the industry (you’re missing the point), but I hope we at least opened a few people’s eyes to the world of testing.

If you’re reading this, this post probably isn’t for you – but get your coworkers (just the one’s you care about should be fine) to look beyond their little part of the world to learn. It’s a big world of testing out there.

Why I Write and Speak

I’m planning to give an internal presentation this week on what I do. I’ve been in my new job for three months, and although I’m still learning, I have enough of a routine that I’d like to share it with a few peers across the company to get feedback and ideas. Most of what I do is my job (that’s probably important, right?). I also still do a lot of cross company stuff (that’s probably pretty important too!).

With my remaining hours, I do “other stuff about testing” (I’ve been thinking about calling this “Project OSAT”, but it’s probably better to just call it “other stuff”). This includes writing books, contributing to other books, writing articles and blog posts, interacting on twitter, and participating in the overall testing community as much as I can find time for.

Sometimes, colleagues ask why I write, or why I speak at conferences. It’s not fame, and it’s certainly not for the money. It’s for ONE reason, and probably because of my lack of sleep due to 4:30am world cup matches, I’m going to share my motivation with everyone.

I write and speak because I’m lazy and cheap.

I suppose I should explain…

How We Test Software At Microsoft was written for one main reason. I talk to a LOT of people and companies about testing – many want to know Microsoft’s general approach to testing. I gave many of these companies the same answers. Over, and over and over again. Finally, I decided (with the help of a few colleagues) to write it all down and sell it for 25 bucks a pop (street price). There was just no way I could keep my sanity and continue to tell people about what testers do at Microsoft. Most of the articles and blog posts I’ve written have been for the same reason – I’m too lazy to give the same answer to multiple people, so when I start hearing the same question too many times, I write it down somewhere (NOTE: this is often also a good heuristic for automation).

Let’s not forget cheap. I hate paying to attend conferences (even if it’s technically my employer’s money). I do like to meet with other testers and talk shop (aka Project OSAT), but if I can get into the conference for free, I’m all over it (sometimes I even manage to get flights or more covered). So I submit proposals to a few conferences a year, and I typically get asked to speak at a few more. I do try to do a good job when I’m there (despite my laziness, I have a high bar for personal quality), but I go to conferences mostly because I get in for free.

Just because I’m lazy and cheap doesn’t mean I don’t care. I love writing, and I love speaking about testing and sharing what I’ve learned. I have worked, and continue to work extremely hard at both of these endeavors (see the note about my high bar for personal quality above). In fact, I’ve never taken money for any writing or speaking (some of probably think this is dumb, but it’s true). Way back when I first started writing for Better Software, I did this because not getting paid was easier than trying to decipher the company moonlighting policy. These days, I’ve found that I actually can get paid for most of this stuff, but I still don’t bother with it. When I consider how much I enjoy writing and speaking, doing it for free seems like the right thing to do (as long as you give me something free too – I’m still cheap).

This week’s conference

Every year, Microsoft’s Engineering Excellence group holds an internal conference – five days of talks, panels, booths and receptions (in other words, it’s just like any other conference, except only Microsoftees are invited). This is the first year since 2004 where I’ve been at the conference and not in the EE group. You’d think that it would be a new experience, but somehow I managed to get myself signed up to present in more sessions than I’ve ever been involved in at this event.

But first, on Monday morning, as the opening keynotes were just ending, I was talking to a group of brand new Microsoft employees. I speak at Microsoft’s new employee orientation fairly often, and it’s something I really enjoy. In the afternoon, I was involved in two sessions. The first was called something like “what testers at MS should know about testing outside of MS”. I think a lot of testers in the industry think that all there is to know about testing and software development exists in their itty bitty tiny corner of the world. Because they never look outside of their little corner, they never realize the truth (if you’re reading this, this probably isn’t referring to you). A colleague of mine kicked off the discussion, and I handled the middle section on communities. I talked about many of the forum and discussion sites currently available, including a shout-out to weekend testers. The session closed with a discussion of non-MS testing tools, specifically in the open-source area.

The second talk was a panel discussion on “The Reality of Careers in Test”. I thought the talk had a good setup, when the “baby” of the panel announced that he had been testing for only 11 years (the range went to 24 years). We had just worked the q&a into a groove when it was time to wrap up – it would have been nice to have a longer session, but it went well.

I hosted a two hour session on Tuesday. We took a deep look at one class of tools used by testers at Microsoft and had panels of owners discuss features and answer questions. One sort of cool thing we tried was to display a feed of questions and comments from (an internal twitter-like tool) during the discussion section. I thought it worked out pretty well and will have to think about how I can do something similar at an external conference someday.

In a session today, I hosted a sixty minute roundtable discussion on test careers. The discussions went well (from my point of view at least), and it was fun. Tomorrow I’m invited to a “special leadership” session (no presenting, just listening). I’m going to try to attend the whole session, but I’m probably going to have to break to spend some time on my “day job”.

I’d love to say that next year I won’t give any talks, but I’m pretty sure that’s not going to happen. I like to take what I know, combine it with my experiences, and try to help people learn something new or get better at what they’re already doing, so I guess there’s no avoiding it for me.

But that’s ok!.