The first thing I did was drain the gas tank. I couldn’t recall if I added fuel stabilizer, but after that long, I can pretty much guarantee that the fuel was bad. I took the bad gas to the hazardous waste site (open on Sunday from 9-5!), picked up some new fuel, headed home, and gassed the Duc up.

I had the battery on a battery tender, but was still slightly surprised that it still had some starting power in it. Unfortunately, the engine just wouldn’t turn over. I double checked the fuel line (clear) and then pulled the spark plugs. The plugs were a little dirty, so I swapped them with a spare set from the toolbox.

Still nothing.

Sitting on the bike, I took a moment to think through how the engine worked. The starter was working, gas was flowing, but the engine wasn’t starting. Fortunately, I have a carbureted engine, and know what all (or most) of the engine workflow. There could still be bad gas still in the system, or there could be a problem with the carburetor. But neither of those seemed likely. I tried starting it one more time, and the engine just wouldn’t kick in.

While thinking through it some more, I noticed that I forgot to reattach one of the spark plug caps. I reattached it and…still nothing.

But – the behavior (i.e. engine sound) was identical with and without the spark plug cap attached – which pretty much guarantees that the spark plugs weren’t firing. I took them out one more time, cleaned them, and this time, checked the gap. For some reason, my spares were gapped really narrow (hint – always check the gap – even on brand new spark plugs selected for your vehicle). I widened the gap a few millimeters to spec, put them in and…

Vroom!

I immediately grabbed my helmet and gloves and went for a spin, and the bike ran great. No stalls, backfires or stutters. It probably still needs some more air in the tires and an oil change, but it sounds and runs just like a Ducati should.

In the end, this was just another debugging and diagnostic problem – much like the problems I face almost every day. The key points to remember are:

• Know the system, and think about the system. When software (or Ducatis) fail, think through the entire system to note where failures may be occurring
• Observe what’s going on – Products (and engines) fail for a reason. Chances are that there are unnoticed clues to the behavior you are seeing, so remember that anything you see may be helpful.

A few weeks ago, I was talking to a colleague about team responsibilities. As an exercise, he was trying to come up with a two word action that described what a discipline was responsible for – the action that you can count on. For development, we agreed quickly on ‘quality code’. There’s certainly more that a developer does, but given the two word action requirement, I can live with our conclusion.

The interesting conversation occurred when we discussed test. My initial answer was ‘provide information’ – is accurate – but not the right answer (for me, at least). I love and hate the notion of tester as information provider. We do generate data (and ideally actionable data) as a side effect of our testing, but that description makes it appear as if test has no power or responsibility for decision making – which I also find wrong. We are not gatekeepers of quality or safety nets, and we’re probably not going to block a release, but I think that testers need to do much more than passively provide information.

My colleague countered with a phrase completely on the opposite end. He proposed ‘sign off’ – that the responsibility of test was to ‘sign off’ on the product (and in order to sign off, we’d generate information, make decisions, etc.) As you can imagine, I didn’t like this description. I’m not against test weighing in on the sign off decision (or any other decision), but I dislike the idea of sign off being the primary responsibility. (Note – Catherine Powell has a nice article on the Decision Safety Net on her blog)

I don’t have a great answer yet for the responsibility of test. I like the idea of the role of test as an accelerant of quality – most of what I do has the end result of improving efficiency of and effectiveness of test and development work. ‘Accelerate Quality’ almost works for me, but I can’t say it’s the two word action that a tester should be responsible for. I’ll figure something out, but I’m open for ideas if you have them

I don’t have a time machine, but I think one positive note from this thought exercise is that I don’t think many (experienced) testers would list the primary action of a tester as ‘write tests’ or ‘find bugs’. At least not too many…

• The Art and Science of Load Testing Internet Applications
• Model-Based Testing for Data Centric Products
• Successful Test Management: 8 Lessons Learned
• Managing User Acceptance Tests in Large Projects
• Architectures of Test Automation
• Testing Rapidly Created Web Sites
• Measuring Ad Hoc Testing
• The Habits of Highly Effective Testers

There are definitely a few interesting topics above, and I’d probably attend all of the sessions if I went to the conference. Unfortunately, it’s too late to register, as the conference I pulled these titles from is nearly twelve years old.

I’ve complained in the past about the lack of new ideas in testing, but despite a massive amount of idea-regurgitation, I think software testing is edging up on a growth spurt. It takes a bit of “cooking” to come up with new Ideas (good ideas, at least), and two of the biggest ingredients seem to be falling into place.

First off, ideas take time to germinate. Testers are beginning to stay in testing longer (you’ll still see half or more of conference attendees with a year or less of testing experience), but anecdotal information, along with my testing spidey-sense tells me that we have more testers staying in test longer than we did a decade ago. With the experience, comes the ability of these testers to draw on enough experience and experiment long enough to bring new ideas to fruition.

Big ideas are often collisions between smaller ideas. You little tool or approach may not be much on it’s own, and my idea for short-circuiting the fizzbazz is a novelty at best. But when we discover each other’s ideas, a new idea may emerge (use the fizzbazz within your approach to make magic happen). In order to enable these collisions of small ideas, we need people with ideas – but we need a network to get those ideas to happen. The degrees of separation and the size of networks is massively larger than it was a decade ago. I think testing is ready to foster idea collision on a massive scale.

I could go on, and on…and I will – but not now.

I’ll be giving a talk on Where (Testing) Ideas Come From as part of the Eurostar Virtual Software Testing Conference. Register for free (‘cause I’m all about making money :}) and hear more of my crazy ideas.

Here’s the short version for those already bored with the topic. Roles that testers play on teams vary. They vary a lot. You can’t compare them. That’s ok, and (IMO) part of the growth of the role. I find it disturbing and detrimental when testers not only assume that their version of testing is “the way”, or that some roles (or people in those roles) do not qualify as testing.

And now for the longer version… The recent test is dead meme (which interestingly, won’t die) brought to light (in semi-dramatic fashion) that in some situations, some traditional “test” roles don’t exist anymore. It wasn’t originally phrased that way, but if you looked under the covers, that’s all that was there. I’m still surprised that so many people got stuck on the three-word catch phrase and couldn’t see the value in the statement. But if they did, I suppose I may not be spitting out this blog post.

Last year, I had a surprisingly popular post about My Job as a Tester. I’ve changed roles since then, and I’ve been thinking about an update, but for a variety of reasons, I’m just not ready yet. The biggest reason is that although I’ve been on the team for five months now, my role is still evolving. Once it settles into a bit of a groove (and as other factors resolve), I’m sure I’ll post a recap.

Recently, I’ve been working a lot on pieces of implementation of test infrastructure for my team. Although I’m still heavily involved in testing strategy and test “stuff”, the goal of most of my current work is to enable good testing. Since I sometimes describe my role as an improver of tests, testers, and testing, I’m still on target with my own vision.

While reflecting recently on what I’ve been doing vs. what [anyone else] does as a tester (and catching up on reading, I pondered the fact that what I’m doing now isn’t really testing as “traditionally” defined (whatever that means). However, what I do is making testing better – but am I more of a “productivity engineer” than a tester?  Brent Jensen has this description:

A tester’s job is to accelerate the achievement of shippable quality

By that definition, I suppose I’m right on the money. But I know there are people (who will likely tell me I’m damaging the craft, or that I’m mean to them) who don’t call what I do testing – I’m cool with that. I still like my job and my business card still says “Tester”.

By far, my favorite favorite thing to do as a tester is design tests. I love the challenge of crafting a suite of tests that enable team members to make well-informed decisions about product quality (at least that’s the plan). Testers in this role may be part of a whole-team approach where they have a test/quality focus, but have shared team goals. Or, there may still be “devs” and “testers”, but the wall between the two is minimal, and everyone works together (most of the time, at least) to make sure the product achieves both shipping and quality goals. Brent’s definition works pretty well for this role too.

Design overlaps execution. The Think-Design-Execute loop is tight in good testing – this is true whether it’s entirely, partially, or non-automated (or inversely, entirely, partially, or non-manual).

Which leads me to two test roles that, while they definitely exist, I could say they’re dead to me. But they’re not really dead – I know that. What they are is more…irrelevant. Given what I do, where I do it, and a smattering of other context indicators, two test roles are off of my radar.

The first is the test-automation only role. I think the role of taking manual test scripts written by one person and then automating those steps is a bad practice. I know some people like to do this stuff, but I think it’s a waste of time. What you end up with are tests that either should have been automated in the first place, or tests that should not be automated. Fortunately, while I acknowledge that these roles exist, I’m happy to  work in a world where these roles do not exist.

For lack of a better term, I’ll call the final role “waterfall-tester” – even though I know this role exists at some (fr)agile shops as well. This is the when-I’m-done-writing-it-you-can-test-it role. Test outsourcing is the most common manifestation of this role, but it exists anywhere where testers only provide value at the end of the cycle. I know great fantastic testers who love this role, and I’ve been in this role myself in the distant past. Today, however, I don’t want to think about testing something where my contribution hasn’t been part of the end-result from the earliest stages. Again, while I fully acknowledge that testers live in this role, I’m happy that it isn’t part of my testing world.

In the end, I’m not exactly sure what this means to anyone but me. As I’ve mentioned (and tweeted) before:

Which really says, that in nearly a thousand words, I’ve (once again) told you nothing new.

Earlier today, Ron posted the following about the Microsoft Tester Center: So Long, Tester Center.

There’s nothing I can say that Ron didn’t say – it was a fun effort – just hard to do without any full time support (and as our individual jobs got more complex and demanding).

The good news, is that I (and many others) remain passionate about sharing some of the cool ideas and innovations coming out of Microsoft. The smartest testers in the world work here, and if you pay attention, you may just get some unique sneak peaks into some of the coolest testing approaches, tools, and ideas in testing you’ll ever see.

Normally, I don’t blog about news stories, but on the way to work this morning I heard an interesting discussion on a local talk radio station. It turns out that there are people up in arms over the purchase because using robots in the warehouse puts people out of jobs. To be clear, I don’t like people losing their jobs either – but bear with me for another paragraph.

The arguments continued with comments like, “The robots can’t do all of the warehouse work – sometimes you need to use your brain to find a misplaced item”, or, “You will still need people to verify with their eyes that the right items were selected”. Others countered with comments like, “The robots can work 24 hours a day”, and, “You’ll still need people to program the robots and give them directions”.

I laughed out loud in my car as I realized that these people calling into the radio station were having the same discussion many people have about test automation.I won’t rehash or expand, but I will summarize with two blurbs of less than 140 characters each:

Since I didn’t use the word test in those tweets, I think my comments apply to Amazon’s (probable) warehouse changes as well.

I was happy to hear that a few callers recognized that robots (and automation) can be used to solve left-brained monotonous work – and do it pretty well. The future of work is (IMO) creative work**, and (again, IMO), automation (and robots) are what we need in order to enable us to find the time and insights to develop our creative thoughts into game-changing innovation.

And that sure beats doing the boring shit.

**Daniel Pink wrote a whole book on this concept:  A Whole New Mind: Why Right-Brainers Will Rule the Future

Jake had barely taken a sip of his steaming coffee when he saw that thirty-two of the automated tests failed in last night’s test pass. “Crap, I’m slammed today”, thought Jake, “I don’t have time to look at thirty-blanking-two failures”. Without a second thought, Jake clicked the ‘re-run failures’’ button on the web page that displayed results and turned his attention back to his coffee. After finishing his coffee and filling a second cup, Jake was happy to see that twenty-five of the failing tests now passed. “Must be a flaky environment”, thought Jake as he took a big swig of coffee and got to work investigating the seven remaining failures.

 

A few weeks later, Jake was sitting in in a meeting to go over a few of the top the live site failures reported by customers and the operations folks. Ellen, the development manager was walking through the issues and fixes, and throwing in a little lightweight root-cause analysis where appropriate. “These three”, she began “caused a pretty bad customer experience. When we first looked at the errors, we figured it had to be an issue with the deployment environment, but we discovered that we could reproduce all of these in our internal test and development environments as well.” Jake’s stomach sunk a bit as Ellen continued. “It turns out that although the functionality is basically broken, it will work some of the time. I guess our tests were just lucky.”

In some versions of this story, Jake steps up to the plate and takes responsibility. In other versions, he merely learns a lesson. In a few versions of the story, Jake calls the whole thing a fluke and goes through the same thing later in his career.

The point of this story is simple. Every test failure means something. The failure may mean a product failure. It may mean you have flaky tests. When you start to assume flaky tests or environments, you’re heading into the land of broken windows and product failures you could have found earlier (actually, you probably did – you just ignored them).

Great testers rely on trustworthy tests. The goal is that every failed test represents a product failure, and any tests that fall short of that goal should be investigated and fixed – or at the very least updated with diagnostic information that lets you make a quick confident decision about the failure. Relying on test automation for any part of your testing is pointless if you don’t care about the results and look at failed tests every time they fail.

Yes, I know. Your situation is unique, and you have a business reason for ignoring failed tests. My first response when I hear this claim is that you’re probably wrong. Probably, but not definitely – but don’t let flaky tests get through your reality filter. Otherwise, you’ll be sitting in Jake’s shoes before you know it.

I spent a lot of time reading on the beach (and fell in love with my Kindle again). I made it through four books (one was a re-reading of a cheesy LOTR rip-off I originally read thirty years ago, but it was fun).

First on my reading list was The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries. Quite a few colleagues recommended the book, and I thought it was good. Two things made this a good (but not great) book for me. The first is that there’s not really anything new in the book (at least for those who have paid passive attention to lean and agile movements over the past few years). Even so, the abundance of stories about iterative practices in a variety of industries more than make up for the repetition. I was also a bit put off by what came across as author arrogance as well. There was just a bit too much self-promotion sprinkled throughout the book for me to fully enjoy it. YMMV, so I’d still recommend this book.

Next up was Influencer: The Power to Change Anything by Kerry Patterson. We have an internal course at MS based on the book. I haven’t taken the course, but I’ve heard enough about the concepts to want to know more (for sick reasons beyond my control, I read a lot about organizational change and organizational learning). The gist of this book (like many others in the category) is that you can solve problems by changing behavior. The value in the book is in the model they use to describe the different ways one can enable behavioral change. I liked that the model divides the concepts into Motivational and Ability groupings – it reminds me of what Michael Lopp calls “The Skill and the Will”[1]. I read Switch: How to Change Things When Change Is Hard earlier this year, and think it’s a bit better book if you’re just going to read one book on organizational change, but the two books complement each other quite well.

The final book I read on the trip was Monday Morning Mentoring: Ten Lessons to Guide You Up the Ladder. First off, I’ll admit that I’m a big fan of business fiction – especially when done well. There were no plot twists in the book, nor anything radical, but for me, it was a page turner (or button pusher in my case). The story documents progress (and setbacks) as a young exec meets with a business guru over the course of ten Monday morning mentoring sessions.

[1] Managing Humans: Biting and Humorous Tales of a Software Engineering Manager – is a fantastic book on management and IMO, essential reading for people managers.

DISCLAIMER: The above links are all amazon associate links (meaning that if you click the link and buy a book, Amazon credits me a few cents. I pull down the associate links automatically, but if you want to buy the books straight away, just go to Amazon directly and search for the above.

The first incident was a common one. My son asked me what the circumference of the earth was (ok – he didn’t use the word ‘circumference’, but that’s what he asked). Like a trained seal, I grabbed my smart-phone and looked it up. This was stupid, because a) I knew the answer, and b) even if I didn’t, I know enough about distances between locations and longitude to come up with a fairly accurate approximation.

Just a bit later in the morning, I had a short interview with a journalist from my university – apparently they’re hard up for stories and are reaching out to alumni for gossip. At one point, the journalist asked something like, “What advice do you have for students today?”. I gave a long answer that said (more or less), “A university isn’t a vocational school. Although you may get caught up in the rat race of learning facts and writing papers, to be successful in the ‘real world’, you need to know how to learn.” I also (once again) credited my Methods of Musical Research course and Dr. Snedeker for teaching me how to learn and for fueling my own passion for learning.

Less than thirty minutes later, I received an email response to a verbal request. I asked someone to research a topic and see if they could find any interesting papers that could foster additional discussion on the topic. They replied with a link to paper that shows up as the top link in Bing when searching on the exact phrase as the topic I presented. I expected my colleague to look at academic papers, and perhaps explore the references in those papers, or to find experts in the field and see what they wrote. I expected them to research…but I guess to them, ‘search’ was enough.

Perhaps I’m just being an old fart, but the serendipity of these three events launched a thought that scared the crap out of me. In these days of search engines and Wikipedia, I wonder if anyone knows how to think anymore. Knowledge is much more than learning or regurgitating facts. The learning that happens when you move knowledge from something you don’t know to something you do know is trivial compared to when you add discovering what you don’t know you don’t know to the top of the stack (1). I’m scared that too many people (not just testers) are happy with on-demand fact finding rather than true learning and discovery.

It could be just me – perhaps I spend so much time reading and trying to learn because that’s what works for me, and that on-demand learning is the true future. But if that’s true, I worry what kind of future that will be.

(1) For more on this theory of knowledge acquisition, read Philip Armour’s original paper on the topic here, or my takes, here, here, and here.

If you haven’t read this book, I bet it’s because the title scared you away. You’re probably thinking, “I’m not a consultant, this book is irrelevant”. For those thinking that, or for any others reading right now, I’ll tell you this. I have read (and in some cases studied), over 40 books on leadership – and this is one of the best I’ve ever read on the subject.

When people get to “Senior” roles at Microsoft, one of the things we expect is an ability to influence without authority, work through others, and basically make those around you better. For our introverted tech geeks, this is often a big wall to climb. Inevitably, when one of these Senior non—manager folks tracks me down and asks, “How can I get better at influence?”, or “How can I get others to accept my ideas?”, I hand them a copy of this book (I’ve done my best to pad Mr. Weinberg’s royalties with the dozen or so copies of this book I’ve purchased over the years). You see, being a good consultant, and being a good leader are pretty much the same thing. Some of my takeaways are below (adapted heavily to my usage over the years).

Good consultants know how to define the real problem at hand – they see what’s there, and what’s not there, and come up with a solution (or multiple solutions). I would hope, that if you’re a leader, you’d do the same.

Good consultants are humble – they don’t pretend to know it all, and know when to admit when they don’t. Your fee (or your title if you’re not a consultant) doesn’t mean you’re always right. It does mean, however, that you always need to help the team improve.

Establish and maintain trust. I can’t say enough about how much I value trust as a leadership attribute. The chapter on trust is critical for any leader.

Worry about the accomplishment, not the credit. This one rings home at MS, where our (ahem) review system often drives people to focus on credit rather than outcomes. The punch line, is that in my experience, the weight of the accomplishment always outweighs the weight of who did what. Just worry about making your projects successful, and the rewards will follow.

And given the hundred or so bullet points of wisdom sprinkled throughout the book, I’d say the meta-takeaway is that you have a good leadership toolbox. If something doesn’t work, doing it again louder won’t help – do something different. Use another leadership tool, or experiment. Nobody has time (or money) for a one-trick pony.

Now – to be fair, several points in the book are specifically for consulting – but if you read this book looking for leadership advice, that’s exactly what you’ll find (and you’ll barely notice the n/z sections).

What are you waiting for – go learn how to be a leader.

I try to read a lot about testing in blogs, articles, books etc. A few days ago, I came across this quote, and it struck me in an odd way.

“Commonly, test automation involves automating a manual process already in place that uses a formalized testing process”

The source doesn’t matter, as it turns out that sentence was copied directly from the Wikipedia article on Test Automation. I’ve been at Microsoft for a long time now, and although I do try to stay connected with testing outside of the Borg, sometimes I notice that *my* view of a subject is quite different than what’s “commonly” understood.

Or, maybe not – but let me try to explain my concerns here a bit more.

I’m all for saving time and money, but I have concerns with an automation approach based entirely (or largely) on automating a bunch of manual tests. Good test design considers manual and computer assisted testing as two different attributes – not sequential tasks. That concept is such an ingrained approach to me (and the testers I get to work with), that the idea of a write tests->run tests manually->automate those tests seems fundamentally broken. I know there are companies that have separate test automation team that do exactly this, and I think it’s a horrible approach.

Let’s use the Numberz Challenge as an example. I know right away that I’m going to perform some manual tests (e.g. make sure the app can launch and close (original version had a bug here), verify look and feel, color choices, etc.). These sorts of tests could be automated, but I generally don’t for a few specific reasons.

• The oracle for look and feel is difficult. Automated testing based on comparing screen colors or app dimensions is fragile. A pair of eyes for a few seconds every once in a while is much cheaper. There are, of course, exceptions, and if you’re convinced you can pull this off, just make sure you calculate the time spent investigating failures (and potential false positives) into your ROI calculation. I know many teams who have successful UI automation systems, but many more who have a bunch of test auto-crapation.
• Most systems I work on – and certainly everything that has a UI, gets at least a tiny bit of usage from the product team or a small pilot group before rolling out widely. If the screen is pink, or the font somehow changed to Comic Sans, it will be noticed without the need for an automated test.

When I look at something like Numberz, I also see where I need to use the power of a computer to answer some of the questions I have. For example, I know I need a large enough data set to be confident of the random algorithm, and I know I need extensive pattern matching to ensure that the next number is never predictable. Doing this manually is impossible (or a waste of time depending on what you try to do).

Now I imagine this scenario in the write-test-then-automate-it workflow.

   1: Test Case Steps:

   2: 1) Launch App

   3: 2) Press the Roll! Button 

   4:  

   5: Verify:

   6: 1) Ensure that none of the numbers is less than 0 or greater than 9

   7: 2) Ensure that the total field is correct

   8:  

   9: Repeat this test at least 10 times

Now, the “automator” comes along and writes this test:

   1: Loop 10 times

   2:   App.Launch (“numberz.exe”)

   3:   Button.Click(“Roll”)

   4:   // Validate numbers are within range 

   5:   // Validate correct total

   6: End Loop

From what little context I have from the interwebs, this appears to be a common scenario — but we’ve missed the critical aspects of testing this app (testing randomness / predictability). What we haven’t done at all in this situation is test design. To me, test design is far more holistic than thinking through a few sets of user tasks. You need to ask, “what’s really going on here?”, and “what do we really need to know?”. Automating a bunch of user tasks rarely answers those questions. (Note – it does answer some questions, so if you have a good system and tests you can trust, by all means don’t stop what you’re doing).

Where I think I have my big disconnect is in the definition of test automation. When I think of test automation, I don’t think of automating user tasks. I think, “How can I use the power of a computer to answer the testing questions I have?”, “How can I use the power of a computer to help me discover what I don’t know I don’t know?”, and “What scenarios do I need to investigate where using a computer is the only practical solution?”.

Perhaps test automation is purely the automation of manual tasks, and I’m attempting to overload the word. I know some folks prefer the term “computer assisted testing”, and I suppose that’s fine too.

To me (and I’m sure I’ve used this line before), it’s just testing. But please stop thinking of test automation as the step that follows test design, and start thinking of test design first.

But it’s easy to go overboard with skepticism. Time and time again, I hear good testers apply their skepticism broadly and liberally. Some (paraphrased) quotes I’ve heard recently include:

• “Model-based testing is interesting, but it doesn’t work in a lot of places”
• “I’m skeptical of static analysis tools – sometimes they have false positives”
• “Metrics are evil, because someone may use them to measure people”

I agree whole heartedly with each of these quotes. However, I worry that folks are throwing the baby out with the bathwater. Model-based testing (just an example), is a wonderful test design technique for stateful test problems. Although occasionally someone will screw up the value of MBT by claiming that it’s the only test approach you’re ever going to need, it’s just another technique (and the perfect technique given the proper context). Static analysis tools are also awesome, but aren’t perfect It’s good to measure some things too, but sure, one can screw it up.

I’m trying to think of anything in my life that works perfectly in every situation, but I’m coming up empty. I run into situations nearly every day where someone has a good idea that will obviously work most of the time – but not always. Given these situations, we could just send them back to the drawing board telling them, “I’m skeptical of your approach, because it won’t work in situation z”, but it’s probably a better idea to have a conversation about the limitations, understand where and when the approach may fail, and discuss mitigation or workarounds. Instead of throwing out the idea of running static analysis tools because of the potential false positives, discuss the false positive problem. Find out what causes them. Tweak the configuration. Do whatever you need to do to ensure the value of the approach.

Over the years, I’ve found value in some pretty stupid approaches. It seems that we should be able to find more value from some the ideas frequently discounted.

Even if we’re skeptical.

In my quest for knowledge, I discovered a few interesting things.

I discovered the Spinellis book, Code Quality: The Open Source Perspective. I haven’t read this yet, but I ordered it, and looks like it will be interesting.

I found an article on coupling metrics on the IBM site. The reading here was good, as I discovered they have a series of articles on code quality.

I found a few references to analysis tools for JavaScript (some lint-like stuff)

Over in the high brow world of the linkedin groups, I saw a bunch of people referencing code coverage as an indicator of code quality.

Nothing earth shattering, but some good information, and enough to get me thinking about what *I* think code quality is – and what it is not. I think it may be impossible to define code quality in a way that everyone will agree with – but that doesn’t mean I shouldn’t try.

We have some metrics that can give us an idea about code quality. Quality code doesn’t have any compiler warnings (ideally at the highest warning level), and it’s clean from static analysis errors and warnings. Complexity metrics such as cyclomatic complexity (basically, the number of paths through a function), or dependency metrics such as fan-in and fan-out can also be indicators of code quality.

But – let’s make it clear that while I’m a (big) fan of those indicators, that’s all they are. Running clean with static analysis tools does nothing to ensure code quality – or product quality either (I ranted a bit about this a few months back). In fact, I guarantee you that I could write (or show you) a piece of software that compiles with the highest warning levels set and passes any static analysis tool with flying colors…that still sucks. 

One thing definitely unrelated to code quality is code coverage. All code coverage tells you is what percentage of your code you haven’t tested at all. If you squint and have a few drinks, you can pretend a little that high coverage rates from unit tests prove code quality…but once you sober up you’ll realize that all you have are unit tests that exercise some amount of your code.

But we can go beyond code analysis tools. Testability (can I observe and control program flow), and Maintainability (“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.” ref) are critical to code quality. Given a good reviewer, Thom Holwerda’s comic is also spot on – the only valid measurement of code quality is WTF’s per minute.

As I think about this, I find my mind wandering back to Pirsig. Quality and care are two different ways of looking at the same thing. Quality comes from care, so perhaps what I want to measure is the care put into coding. The fact that a developer wanted to compile at a higher warning level or turn on more static analysis checks, or that they cared enough to get the testers input on testability, or a peer’s advice on maintainability may be the best ways to measure code quality of all.

So now, perhaps, I’m in search of a way to discover and measure care.

Historically, most of my blog posts come from experiences or inspiration from my day job (despite those who seem to think that every snide comment I make is about them, my snide comments are almost entirely about people I work with). While I’m sure that will continue to be the case, I’m still pretty deep in ramp up mode, and it turns out that the bulk of the work I need to do right now is solving a variety of test infrastructure and code issues. While I find this sort of work exciting to perform, I find it dreadfully boring to write about (apologies to all those who enjoy writing about these things).

But I do enjoy writing, and even as I write this trivial little blurb (a blurb that will turn into a blog post in about five minutes), I realize that I have a huge backlog of items not contrived from my day-to-day work that I can draw from in order to satisfy my need to write and share a few times a week.

Meanwhile, some chunks of my work are about things that I can’t blog about. In some ways, those are the most exciting. I have plans to share some of my thoughts on this when it’s appropriate, and I think it will be an interesting exercise (and hopefully, interesting reading when I can share). We’ll see what happens.

I wrote 60 blog entries in 2011. Some of the more popular posts (in no particular order) were:

• My job as a Tester
• What is Testing?
• Test Design for Automation
• Numberz Challenge
• Beyond Regression Tests
• R-E-S-P-E-C-T
• Judgment in Testing
• Lost in the weeds

I did a bit of speaking in 2011 (although, as usual, more than I planned). I was a pnsqc, at Intel in Israel, and at the Swiss and German Testing Day conferences. I also attended the second Writing About Testing conference (and had a great time).

Next year, I’m trying (again) to speak and travel less, blog about the same amount, and keep having fun working on the Xbox team. For everyone who’s read or posted a comment, thank you for supporting my silly addiction to writing. I hope I’ve given you something to think (or complain) about.

Best wishes for the holidays and 2012.

I’m a Senior Tester, don’t you know

Been testing software for five years or so

I got bills to pay and need a better job…

Looking for jobs with ‘Senior’ in the name

Gotta’s admit, they’re all pretty lame

Looks like I’m over-qualified…

Someone (non-Microsoft) asked me recently what a “Senior Tester” does. I told them that I knew in the context of Microsoft, but really had no idea if the term meant anything in the industry. After a quick (and painful) trip to a jobs website, I was able to confirm that the ‘Senior’ title doesn’t carry a lot of weight. Consider some of these quotes – all from positions advertised as “Senior Tester”, or “Senior QA Tester (on a side note, wtf is a QA Tester?):

• With minimal direction writes scripts and executes test cases during functional/system/integration testing.
• Modify existing test plans and testing information to correct errors, to adapt to new test scenarios, upgraded interfaces and performance enhancements.
• Monitor bug resolution efforts and track successes.
• Participates in analysis and design walkthroughs, as well as team meetings
• The Senior Tester will work extensively with the Test Lead to provide test estimates
• Complete knowledge of QA Best Practices and the full Testing SDLC

Let me rant on the last bullet point first. Actually, never mind – anything I can say is something you’ve probably already thought.

As for the first five bullet points, are those really considered “Senior” activities? I was surprised that being able to cross the street by themselves or washing their hands after using the bathroom weren’t required attributes. Then again, I’ve been sheltered in the world of Microsoft for the past 17 years (and as a noob in a startup for 2 years before that), so my world view of testing has limitations.

So, let me talk about what I do know. Here at the evil empire, we have standard titles for all roles (more details on test titles are here). As I mentioned in the link (which I know you haven’t read yet), we have a “Senior SDET” title at MS, and this is where we expect testers to exhibit some leadership skills (they learn how to find the bathroom by themselves at a much lower level). To be clear, leadership does not mean management. ‘Senior’ is the minimum level where we like people (who are interested) to move into management roles, but we have hundreds of non-managers at the Senior level (about 8% of the total test population if you’re curious).

My random search for Senior test roles wasn’t entirely in vain. Consider the following:

• Experience with leading testing efforts on integrated teams and developing and implementing sound testing methodologies
• This is a senior role that includes evaluating existing automated testing process and recommending improvements

Now those seem a little better – now the role requires some leadership and decision making, so let’s look a bit closer at these a few other attributes I expect to see in Senior testers at MS.

• Decision Making – senior testers make confident decisions, and do not rely on consensus for making (most) decisions.
• Influence – Dwight Eisenhower said, “Leadership is the art of getting someone else to do something you want done because he wants to do it.”  Senior testers make their team better through influence rather than mandates – they are the point guard of the test team.
• Network – senior testers use their network to find answers (or knowledge), and connect their teammates with their network (i.e. tester matchmaking)
• Credibility – senior testers don’t proclaim they’re a leader, and no title demands respect. Leaders know that respect as a leader comes through establishing credibility with their team and peers. For me, the moment you demand respect as a leader is the moment you lose my respect as a leader.

That’s not an exhaustive list, and each of those points could be (and may be) a post unto its own. But that’s what ‘Senior Tester’ means to me (In my sheltered life). Undoubtedly, your definition is different, and that’s fine. Regardless of where your opinions lie, I hope this provides some food for thought on your own views of what ‘Senior’ means.

Admittedly, I’m still in the honey-moon phase, but I’m having a great time. But life just doesn’t seem to be as fun without a surprise here and there, and this career move wasn’t without exception. More on that later – but let’s first consider a bit of career advice.

One tidbit of career advice I often give is “The 3 Ps”. When you’re looking for a job (or evaluating your current position), you need to consider the Product, the People, and the Person.

• Consider the Product – You’ll be more motivated and excited about your work if you have interest in the product (or the technical challenges regarding that product). On the corporate applications side of Microsoft, I honestly think that Lync (my old group) is freakin’ cool, but there’s no understating the sexy coolness of Xbox and Kinect either.
• Consider the People – Nobody wants to work with a bunch of jerks. I don’t want to work with people who don’t care about their jobs. I happened to know dozens of people working on XBox (I worked with many of them on Windows 95 & Windows 98), and all are great people. When I was considering the move, I chatted with people on the team that I didn’t know – and they all had this vibe of loving what they do. To me, that’s the sign of a good team to work on.
• Consider the Person – We all have to work for someone. Take time when considering a new position to make sure that whoever you report to (and ideally, their manager) are on the same page as you; and will support you and help you grow.

To me – the Person part of the equation is critical. I’m in a rare position at Microsoft – we don’t have many testers at my level who do not manage teams. Managing someone like me, and communicating my value (or lack of value) appropriately is a challenge. Because there are few of us extra-senior testers, there are very few managers who have experience using us effectively. I selected my last manager based on this criteria, and when I moved to Xbox, I used the same criteria. It was hard to leave a team where I had the 3 Ps, but the time was right, and I took the time to ensure that I would have the 3 Ps on my new team as well.

Three days into my new job, my manager called me into his office and said, “Welcome to the team – we’re having a re-org”. The details of the re-org aren’t important, but they make perfect business sense, and I’m excited about the change. Unfortunately, the manager I had carefully selected wasn’t going to be my manager any more – and with the upcoming release of the new dashboard, I wasn’t sure where I was going to land until a day or two ago.

So – with one manager on the way out, and another yet to be defined, I fell back on what I seem to do best – I did whatever I wanted. In the land of Xbox, I suppose that could mean a whole lot of Skyrim and Forza, but I dug deep into some testing challenges, looked at engineering systems, got to know people and basically spent a whole bunch of time figuring out how the org and technology work. I once considered a position at a company full of young folks where someone suggested that I spend at least a month or two “doing whatever I wanted” until I figured out what I really wanted to do. My recent experience wasn’t quite as directed as that, but I find it interesting that concept seems to repeat throughout my career.

The good news is that I’ve met with my new boss a few times, and I think we’ll work very well together. We’re still figuring out exactly what I’ll do (which is perfectly ok), but I’m completely comfortable with how he wants to use me, and although he’s still a bit of an unknown to me, I’m pretty confident my 3 Ps are still in place. I’m excited, and a little scared (another bit of good career advice), so I know I’m in the right place.

I see this concept come up frequently, and I’ve noticed a bit of a trend. Teams that follow Just Fix It love it. Teams that prefer to fix bugs later are sure that the concept won’t work for their team, and that they need the curve and tracking data in order to ship their (undeniably unique) product. As a side note, one fun thing I’ve talked a few teams at Microsoft into doing is to pair on bug fixes – when Joe-Tester finds a bug, he Tells Kathy-Developer about the issue, and then the two of them pair-program the fix. I could write a whole post on why this is so cool, but I’ll leave it at that for now.

In short, I’m a huge fan of Just Fix It, but as usual, the totality of overlap of agreement between Chris and I is about 93%.

For example, let’s say on Monday, Alex-Tester notices that the froobnozzle isn’t working. He tells the developer, who fixes the problem immediately. Tuesday, Beth-Tester notices that the froobnozzle doesn’t work when interacting with another part of the system. She tells the developer, who fixes the problem right away. Over the following days and weeks, a lot of problems are discovered with the froobnozzle. The froobnozzle, is, in fact, a piece of crap held together by the 1s and 0s version of spit and duct tape. A bug tracking system let’s you see (I hate to say this out lout…) trends of where errors are. Bugs don’t appear randomly sprinkled throughout a product – they tend to congregate in clumps. Knowing where the clumps are can guide further testing, or risk decisions.

Source control almost mitigates this concern, but unless you have a diligent comment policy for check-ins, you probably won’t be able to differentiate between a “I was adding new code or functionality” check-ins vs. “I was fixing shit I broke” check-ins.

But I won’t go as far as to say you need a bug tracking system. As Chris describes it, and as most people use them, a bug system is really a work-item tracking system anyway. If you track work items on post-its or notecards, bugs should work the same way. As far as trends go, I think a simple tic-mark system would work just as well. When you discover a problem with the froobnozzle, write it on the board and put a tic-mark next to it. When a component gets n number of tics, schedule refactoring time, or do a design review (or both). Alternately, look at the components with the highest number of tics during the retrospective or sprint planning and review those for potential re-work.

But other than that, yeah. Just Fix It!

Note: I would have posted this as a comment on Chris’s blog, but I, for one, find the comment forms on blogspot completely unusable.

Let me outline the problem. I don’t look at email or twitter or my rss feed all of the time – in fact, most of my electronic communication time occurs when I’m compiling / building (I could use that time to have a sword-fight, but I find it better to at least pretend I’m productive). I love to group the “non-essential” stuff together, and when I’m waiting on a build, it’s the perfect time to write email, documents, tweet (or sometimes blog).

The problem is, that when I’m compiling, my 12 cores are pretty busy. This makes reading email, or doing pretty much anything else that would like more than a few time slices of cpu power sort of suck.

I have two computers (a desktop and a laptop), so I have an option of using my laptop for those tasks while the compiler elves are taking over my computer, but given the choice of using a 24” monitor and a full size ergonomic keyboard over a 13” monitor and compressed keyboard, which would you choose? To be fair, I adore my laptop (Lenovo X200s) – but I adore it because it’s small and I can use it anywhere (including on an airline tray table while the monster man in front of me leans his seat all the way back for 8 hours straight). But it’s not a computer I like to write large documents or presentations on.

The solution, as many have already guessed, is to connect to my laptop via terminal server and use it from my desktop machine. The overhead of TS is small enough, that I barely notice a perf problem even at peak memory and cpu usage. The TS session even stretches my desktop to 1920×1080, so it’s a pretty sweet setup. All that is good (and somewhat obvious), but few things in life ever go without a hitch.

Despite how much I wanted to love the setup, I noticed that my connection would drop frequently – and then it would take a minute until I could reconnect. I powered through it for a day or two (I tell myself that I was subconsciously gathering clues). Then finally, I found the clue that led me in right direction. I listen to music on Zune quite a bit while I work (I configured terminal server to play audio on the server machine (my laptop), so I select songs from the TS session on my desktop, and listen through headphones connected to my laptop). So…I noticed that whenever my computer disconnected, I lost audio a few seconds later. The first few times this happened, I (ignorantly) assumed that because I lost the TS connection, I also lost the audio. Then it hit me that since the audio is playing on my laptop, I should never lose audio…unless the network connection was lost.

Losing the TS connection and losing audio were both symptoms of the same root cause (lost network connection) – the only thing left to figure out was why two computers on the same subnet were losing their connection.

I took a look at power management settings to ensure that my laptop wasn’t suspending, but all was good. My next step was to look at my adapter settings to see if the network adapter was powering down for some reason. Sure enough, there was a setting named “System Idle Power Saver” that was enabled. I disabled the setting, and I’ve only had one dropped connection all week.

This example may sound more like yak-shaving than testing, but I’ve always liked the trouble shooting aspect of testing. Going beyond “here’s a problem”, to “here’s what (probably) caused the problem” is, in my opinion) an important skill for testers.

Oops – my build is done – time to get this posted.