Tooth of the Weasel

notes and rants about testing and quality from alan page

Systems, observation, and motorcycles

May 14th, 2012 by Alan Page 3 comments »

Our family spent a bunch of time this weekend cleaning out the garage and taking care of a variety of long neglected household tasks. One thing I’d been meaning to do for over a year now is to get my Ducati up and running again. Between picking kids up (need a car for that), and riding my bike to work most of last summer, it’s probably been 18 months since I started the thing. Keep in mind, that Ducati’s are fickle machines to start with, but I figured I’d work on it for a while before calling someone to load it on a truck and haul it to the shop.

The first thing I did was drain the gas tank. I couldn’t recall if I added fuel stabilizer, but after that long, I can pretty much guarantee that the fuel was bad. I took the bad gas to the hazardous waste site (open on Sunday from 9-5!), picked up some new fuel, headed home, and gassed the Duc up.

I had the battery on a battery tender, but was still slightly surprised that it still had some starting power in it. Unfortunately, the engine just wouldn’t turn over. I double checked the fuel line (clear) and then pulled the spark plugs. The plugs were a little dirty, so I swapped them with a spare set from the toolbox.

Still nothing.

Sitting on the bike, I took a moment to think through how the engine worked. The starter was working, gas was flowing, but the engine wasn’t starting. Fortunately, I have a carbureted engine, and know what all (or most) of the engine workflow. There could still be bad gas still in the system, or there could be a problem with the carburetor. But neither of those seemed likely. I tried starting it one more time, and the engine just wouldn’t kick in.

While thinking through it some more, I noticed that I forgot to reattach one of the spark plug caps. I reattached it and…still nothing.

But – the behavior (i.e. engine sound) was identical with and without the spark plug cap attached – which pretty much guarantees that the spark plugs weren’t firing. I took them out one more time, cleaned them, and this time, checked the gap. For some reason, my spares were gapped really narrow (hint – always check the gap – even on brand new spark plugs selected for your vehicle). I widened the gap a few millimeters to spec, put them in and…

Vroom!

I immediately grabbed my helmet and gloves and went for a spin, and the bike ran great. No stalls, backfires or stutters. It probably still needs some more air in the tires and an oil change, but it sounds and runs just like a Ducati should.

In the end, this was just another debugging and diagnostic problem – much like the problems I face almost every day. The key points to remember are:

Know the system, and think about the system. When software (or Ducatis) fail, think through the entire system to note where failures may be occurring
Observe what’s going on – Products (and engines) fail for a reason. Chances are that there are unnoticed clues to the behavior you are seeing, so remember that anything you see may be helpful.

3 comments »

Posted in Software Testing

Test Responsibility

May 7th, 2012 by Alan Page 14 comments »

I apologize in advance for yet another exploration of what testers do. More and more, I feel that Brent is right, and Test is a 4 Letter Word, but I feel we (whatever we want to call ourselves) can advance through discussion of our roles and responsibilities.

A few weeks ago, I was talking to a colleague about team responsibilities. As an exercise, he was trying to come up with a two word action that described what a discipline was responsible for – the action that you can count on. For development, we agreed quickly on ‘quality code’. There’s certainly more that a developer does, but given the two word action requirement, I can live with our conclusion.

The interesting conversation occurred when we discussed test. My initial answer was ‘provide information’ – is accurate – but not the right answer (for me, at least). I love and hate the notion of tester as information provider. We do generate data (and ideally actionable data) as a side effect of our testing, but that description makes it appear as if test has no power or responsibility for decision making – which I also find wrong. We are not gatekeepers of quality or safety nets, and we’re probably not going to block a release, but I think that testers need to do much more than passively provide information.

My colleague countered with a phrase completely on the opposite end. He proposed ‘sign off’ – that the responsibility of test was to ‘sign off’ on the product (and in order to sign off, we’d generate information, make decisions, etc.) As you can imagine, I didn’t like this description. I’m not against test weighing in on the sign off decision (or any other decision), but I dislike the idea of sign off being the primary responsibility. (Note – Catherine Powell has a nice article on the Decision Safety Net on her blog)

I don’t have a great answer yet for the responsibility of test. I like the idea of the role of test as an accelerant of quality – most of what I do has the end result of improving efficiency of and effectiveness of test and development work. ‘Accelerate Quality’ almost works for me, but I can’t say it’s the two word action that a tester should be responsible for. I’ll figure something out, but I’m open for ideas if you have them

I don’t have a time machine, but I think one positive note from this thought exercise is that I don’t think many (experienced) testers would list the primary action of a tester as ‘write tests’ or ‘find bugs’. At least not too many…

14 comments »

Posted in Software Quality, Software Testing, Test Careers

New Testing Ideas

April 24th, 2012 by Alan Page 3 comments »

I was checking out test conference programs, and found a list of talk titles I found intriguing (this is a sampling of titles from the conference). I’m curious to know how interesting and innovative you think this conference would be.

The Art and Science of Load Testing Internet Applications
Model-Based Testing for Data Centric Products
Successful Test Management: 8 Lessons Learned
Managing User Acceptance Tests in Large Projects
Architectures of Test Automation
Testing Rapidly Created Web Sites
Measuring Ad Hoc Testing
The Habits of Highly Effective Testers

There are definitely a few interesting topics above, and I’d probably attend all of the sessions if I went to the conference. Unfortunately, it’s too late to register, as the conference I pulled these titles from is nearly twelve years old.

I’ve complained in the past about the lack of new ideas in testing, but despite a massive amount of idea-regurgitation, I think software testing is edging up on a growth spurt. It takes a bit of “cooking” to come up with new Ideas (good ideas, at least), and two of the biggest ingredients seem to be falling into place.

First off, ideas take time to germinate. Testers are beginning to stay in testing longer (you’ll still see half or more of conference attendees with a year or less of testing experience), but anecdotal information, along with my testing spidey-sense tells me that we have more testers staying in test longer than we did a decade ago. With the experience, comes the ability of these testers to draw on enough experience and experiment long enough to bring new ideas to fruition.

Big ideas are often collisions between smaller ideas. You little tool or approach may not be much on it’s own, and my idea for short-circuiting the fizzbazz is a novelty at best. But when we discover each other’s ideas, a new idea may emerge (use the fizzbazz within your approach to make magic happen). In order to enable these collisions of small ideas, we need people with ideas – but we need a network to get those ideas to happen. The degrees of separation and the size of networks is massively larger than it was a decade ago. I think testing is ready to foster idea collision on a massive scale.

I could go on, and on…and I will – but not now.

I’ll be giving a talk on Where (Testing) Ideas Come From as part of the Eurostar Virtual Software Testing Conference. Register for free (‘cause I’m all about making money :}) and hear more of my crazy ideas.

3 comments »

Posted in Me, Software Testing

Secrets of SDET Success

April 18th, 2012 by Alan Page No comments »

In case you missed it, I blogged about testers at MS over at the Expert Testers blog (which is probably a blog you should add to your blog roll if it’s not there already).

No comments »

Posted in Software Testing

Exploring Test Roles

April 4th, 2012 by Alan Page 10 comments »

I’m not quite sure why, but once again I’m writing about test roles. I don’t know of another job in the world where discussions like these are common. On the other hand, I don’t know of a job in the world where people are so passionate about what they do (and don’t do) as part of their role. I’ll chalk it up to the continued growth of the role and see if I can convince myself to finish this and post it before I stop myself.

Here’s the short version for those already bored with the topic. Roles that testers play on teams vary. They vary a lot. You can’t compare them. That’s ok, and (IMO) part of the growth of the role. I find it disturbing and detrimental when testers not only assume that their version of testing is “the way”, or that some roles (or people in those roles) do not qualify as testing.

And now for the longer version… The recent test is dead meme (which interestingly, won’t die) brought to light (in semi-dramatic fashion) that in some situations, some traditional “test” roles don’t exist anymore. It wasn’t originally phrased that way, but if you looked under the covers, that’s all that was there. I’m still surprised that so many people got stuck on the three-word catch phrase and couldn’t see the value in the statement. But if they did, I suppose I may not be spitting out this blog post.

Last year, I had a surprisingly popular post about My Job as a Tester. I’ve changed roles since then, and I’ve been thinking about an update, but for a variety of reasons, I’m just not ready yet. The biggest reason is that although I’ve been on the team for five months now, my role is still evolving. Once it settles into a bit of a groove (and as other factors resolve), I’m sure I’ll post a recap.

Recently, I’ve been working a lot on pieces of implementation of test infrastructure for my team. Although I’m still heavily involved in testing strategy and test “stuff”, the goal of most of my current work is to enable good testing. Since I sometimes describe my role as an improver of tests, testers, and testing, I’m still on target with my own vision.

While reflecting recently on what I’ve been doing vs. what [anyone else] does as a tester (and catching up on reading, I pondered the fact that what I’m doing now isn’t really testing as “traditionally” defined (whatever that means). However, what I do is making testing better – but am I more of a “productivity engineer” than a tester? Brent Jensen has this description:

A tester’s job is to accelerate the achievement of shippable quality

By that definition, I suppose I’m right on the money. But I know there are people (who will likely tell me I’m damaging the craft, or that I’m mean to them) who don’t call what I do testing – I’m cool with that. I still like my job and my business card still says “Tester”.

By far, my favorite favorite thing to do as a tester is design tests. I love the challenge of crafting a suite of tests that enable team members to make well-informed decisions about product quality (at least that’s the plan). Testers in this role may be part of a whole-team approach where they have a test/quality focus, but have shared team goals. Or, there may still be “devs” and “testers”, but the wall between the two is minimal, and everyone works together (most of the time, at least) to make sure the product achieves both shipping and quality goals. Brent’s definition works pretty well for this role too.

Design overlaps execution. The Think-Design-Execute loop is tight in good testing – this is true whether it’s entirely, partially, or non-automated (or inversely, entirely, partially, or non-manual).

Which leads me to two test roles that, while they definitely exist, I could say they’re dead to me. But they’re not really dead – I know that. What they are is more…irrelevant. Given what I do, where I do it, and a smattering of other context indicators, two test roles are off of my radar.

The first is the test-automation only role. I think the role of taking manual test scripts written by one person and then automating those steps is a bad practice. I know some people like to do this stuff, but I think it’s a waste of time. What you end up with are tests that either should have been automated in the first place, or tests that should not be automated. Fortunately, while I acknowledge that these roles exist, I’m happy to work in a world where these roles do not exist.

For lack of a better term, I’ll call the final role “waterfall-tester” – even though I know this role exists at some (fr)agile shops as well. This is the when-I’m-done-writing-it-you-can-test-it role. Test outsourcing is the most common manifestation of this role, but it exists anywhere where testers only provide value at the end of the cycle. I know ~~great~~ fantastic testers who love this role, and I’ve been in this role myself in the distant past. Today, however, I don’t want to think about testing something where my contribution hasn’t been part of the end-result from the earliest stages. Again, while I fully acknowledge that testers live in this role, I’m happy that it isn’t part of my testing world.

In the end, I’m not exactly sure what this means to anyone but me. As I’ve mentioned (and tweeted) before:

Which really says, that in nearly a thousand words, I’ve (once again) told you nothing new.

10 comments »

Posted in Software Quality, Software Testing

So Long, Tester Center

March 28th, 2012 by Alan Page No comments »

Earlier today, Ron posted the following about the Microsoft Tester Center: So Long, Tester Center.

There’s nothing I can say that Ron didn’t say – it was a fun effort – just hard to do without any full time support (and as our individual jobs got more complex and demanding).

The good news, is that I (and many others) remain passionate about sharing some of the cool ideas and innovations coming out of Microsoft. The smartest testers in the world work here, and if you pay attention, you may just get some unique sneak peaks into some of the coolest testing approaches, tools, and ideas in testing you’ll ever see.

No comments »

Posted in Software Testing

The Robots are Taking Over

March 20th, 2012 by Alan Page No comments »

Probably not news to most of you, but the local company that sells everything under the sun just bought a robotics company (Amazon Acquires Kiva Systems).

Normally, I don’t blog about news stories, but on the way to work this morning I heard an interesting discussion on a local talk radio station. It turns out that there are people up in arms over the purchase because using robots in the warehouse puts people out of jobs. To be clear, I don’t like people losing their jobs either – but bear with me for another paragraph.

The arguments continued with comments like, “The robots can’t do all of the warehouse work – sometimes you need to use your brain to find a misplaced item”, or, “You will still need people to verify with their eyes that the right items were selected”. Others countered with comments like, “The robots can work 24 hours a day”, and, “You’ll still need people to program the robots and give them directions”.

I laughed out loud in my car as I realized that these people calling into the radio station were having the same discussion many people have about test automation.I won’t rehash or expand, but I will summarize with two blurbs of less than 140 characters each:

Since I didn’t use the word test in those tweets, I think my comments apply to Amazon’s (probable) warehouse changes as well.

I was happy to hear that a few callers recognized that robots (and automation) can be used to solve left-brained monotonous work – and do it pretty well. The future of work is (IMO) creative work**, and (again, IMO), automation (and robots) are what we need in order to enable us to find the time and insights to develop our creative thoughts into game-changing innovation.

And that sure beats doing the boring shit.

**Daniel Pink wrote a whole book on this concept: A Whole New Mind: Why Right-Brainers Will Rule the Future

No comments »

Posted in Software Testing, Test Careers

Oops, I Did it Again

March 19th, 2012 by Alan Page 3 comments »

Here’s a story I hear often. The names have been changed to prevent the guilty.

Jake had barely taken a sip of his steaming coffee when he saw that thirty-two of the automated tests failed in last night’s test pass. “Crap, I’m slammed today”, thought Jake, “I don’t have time to look at thirty-blanking-two failures”. Without a second thought, Jake clicked the ‘re-run failures’’ button on the web page that displayed results and turned his attention back to his coffee. After finishing his coffee and filling a second cup, Jake was happy to see that twenty-five of the failing tests now passed. “Must be a flaky environment”, thought Jake as he took a big swig of coffee and got to work investigating the seven remaining failures.

A few weeks later, Jake was sitting in in a meeting to go over a few of the top the live site failures reported by customers and the operations folks. Ellen, the development manager was walking through the issues and fixes, and throwing in a little lightweight root-cause analysis where appropriate. “These three”, she began “caused a pretty bad customer experience. When we first looked at the errors, we figured it had to be an issue with the deployment environment, but we discovered that we could reproduce all of these in our internal test and development environments as well.” Jake’s stomach sunk a bit as Ellen continued. “It turns out that although the functionality is basically broken, it will work some of the time. I guess our tests were just lucky.”

In some versions of this story, Jake steps up to the plate and takes responsibility. In other versions, he merely learns a lesson. In a few versions of the story, Jake calls the whole thing a fluke and goes through the same thing later in his career.

The point of this story is simple. Every test failure means something. The failure may mean a product failure. It may mean you have flaky tests. When you start to assume flaky tests or environments, you’re heading into the land of broken windows and product failures you could have found earlier (actually, you probably did – you just ignored them).

Great testers rely on trustworthy tests. The goal is that every failed test represents a product failure, and any tests that fall short of that goal should be investigated and fixed – or at the very least updated with diagnostic information that lets you make a quick confident decision about the failure. Relying on test automation for any part of your testing is pointless if you don’t care about the results and look at failed tests every time they fail.

Yes, I know. Your situation is unique, and you have a business reason for ignoring failed tests. My first response when I hear this claim is that you’re probably wrong. Probably, but not definitely – but don’t let flaky tests get through your reality filter. Otherwise, you’ll be sitting in Jake’s shoes before you know it.

3 comments »

Posted in Software Testing

Recharged, reloaded, and a lot of reading

March 2nd, 2012 by Alan Page 2 comments »

I spent last week lounging on the beaches of Maui with my family. I didn’t stay completely away from work or the interwebz, but after a day or so of sun and fun,I did manage to unwind enough to lose track of days of the week.

I spent a lot of time reading on the beach (and fell in love with my Kindle again). I made it through four books (one was a re-reading of a cheesy LOTR rip-off I originally read thirty years ago, but it was fun).

First on my reading list was The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries. Quite a few colleagues recommended the book, and I thought it was good. Two things made this a good (but not great) book for me. The first is that there’s not really anything new in the book (at least for those who have paid passive attention to lean and agile movements over the past few years). Even so, the abundance of stories about iterative practices in a variety of industries more than make up for the repetition. I was also a bit put off by what came across as author arrogance as well. There was just a bit too much self-promotion sprinkled throughout the book for me to fully enjoy it. YMMV, so I’d still recommend this book.

Next up was Influencer: The Power to Change Anything by Kerry Patterson. We have an internal course at MS based on the book. I haven’t taken the course, but I’ve heard enough about the concepts to want to know more (for sick reasons beyond my control, I read a lot about organizational change and organizational learning). The gist of this book (like many others in the category) is that you can solve problems by changing behavior. The value in the book is in the model they use to describe the different ways one can enable behavioral change. I liked that the model divides the concepts into Motivational and Ability groupings – it reminds me of what Michael Lopp calls “The Skill and the Will”[1]. I read Switch: How to Change Things When Change Is Hard earlier this year, and think it’s a bit better book if you’re just going to read one book on organizational change, but the two books complement each other quite well.

The final book I read on the trip was Monday Morning Mentoring: Ten Lessons to Guide You Up the Ladder. First off, I’ll admit that I’m a big fan of business fiction – especially when done well. There were no plot twists in the book, nor anything radical, but for me, it was a page turner (or button pusher in my case). The story documents progress (and setbacks) as a young exec meets with a business guru over the course of ten Monday morning mentoring sessions.

[1] Managing Humans: Biting and Humorous Tales of a Software Engineering Manager – is a fantastic book on management and IMO, essential reading for people managers.

DISCLAIMER: The above links are all amazon associate links (meaning that if you click the link and buy a book, Amazon credits me a few cents. I pull down the associate links automatically, but if you want to buy the books straight away, just go to Amazon directly and search for the above.

2 comments »

Posted in Leadership, Me

Learning is Dead

February 18th, 2012 by Alan Page 9 comments »

Inspiration for my blog posts often comes from whatever’s evoking emotion in my life (annoyances, victories, gripes, etc.). Three things happened today that ensured that I’d write this post. Here’s the story of a question from a seven year old, a phone call, and an email.

The first incident was a common one. My son asked me what the circumference of the earth was (ok – he didn’t use the word ‘circumference’, but that’s what he asked). Like a trained seal, I grabbed my smart-phone and looked it up. This was stupid, because a) I knew the answer, and b) even if I didn’t, I know enough about distances between locations and longitude to come up with a fairly accurate approximation.

Just a bit later in the morning, I had a short interview with a journalist from my university – apparently they’re hard up for stories and are reaching out to alumni for gossip. At one point, the journalist asked something like, “What advice do you have for students today?”. I gave a long answer that said (more or less), “A university isn’t a vocational school. Although you may get caught up in the rat race of learning facts and writing papers, to be successful in the ‘real world’, you need to know how to learn.” I also (once again) credited my Methods of Musical Research course and Dr. Snedeker for teaching me how to learn and for fueling my own passion for learning.

Less than thirty minutes later, I received an email response to a verbal request. I asked someone to research a topic and see if they could find any interesting papers that could foster additional discussion on the topic. They replied with a link to paper that shows up as the top link in Bing when searching on the exact phrase as the topic I presented. I expected my colleague to look at academic papers, and perhaps explore the references in those papers, or to find experts in the field and see what they wrote. I expected them to research…but I guess to them, ‘search’ was enough.

Perhaps I’m just being an old fart, but the serendipity of these three events launched a thought that scared the crap out of me. In these days of search engines and Wikipedia, I wonder if anyone knows how to think anymore. Knowledge is much more than learning or regurgitating facts. The learning that happens when you move knowledge from something you don’t know to something you do know is trivial compared to when you add discovering what you don’t know you don’t know to the top of the stack (1). I’m scared that too many people (not just testers) are happy with on-demand fact finding rather than true learning and discovery.

It could be just me – perhaps I spend so much time reading and trying to learn because that’s what works for me, and that on-demand learning is the true future. But if that’s true, I worry what kind of future that will be.

(1) For more on this theory of knowledge acquisition, read Philip Armour’s original paper on the topic here, or my takes, here, here, and here.