Author: Alan Page

I spent last week lounging on the beaches of Maui with my family. I didn’t stay completely away from work or the interwebz, but after a day or so of sun and fun,I did manage to unwind enough to lose track of days of the week.

I spent a lot of time reading on the beach (and fell in love with my Kindle again). I made it through four books (one was a re-reading of a cheesy LOTR rip-off I originally read thirty years ago, but it was fun).

First on my reading list was The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries. Quite a few colleagues recommended the book, and I thought it was good. Two things made this a good (but not great) book for me. The first is that there’s not really anything new in the book (at least for those who have paid passive attention to lean and agile movements over the past few years). Even so, the abundance of stories about iterative practices in a variety of industries more than make up for the repetition. I was also a bit put off by what came across as author arrogance as well. There was just a bit too much self-promotion sprinkled throughout the book for me to fully enjoy it. YMMV, so I’d still recommend this book.

Next up was Influencer: The Power to Change Anything by Kerry Patterson. We have an internal course at MS based on the book. I haven’t taken the course, but I’ve heard enough about the concepts to want to know more (for sick reasons beyond my control, I read a lot about organizational change and organizational learning). The gist of this book (like many others in the category) is that you can solve problems by changing behavior. The value in the book is in the model they use to describe the different ways one can enable behavioral change. I liked that the model divides the concepts into Motivational and Ability groupings – it reminds me of what Michael Lopp calls “The Skill and the Will”[1]. I read Switch: How to Change Things When Change Is Hard earlier this year, and think it’s a bit better book if you’re just going to read one book on organizational change, but the two books complement each other quite well.

The final book I read on the trip was Monday Morning Mentoring: Ten Lessons to Guide You Up the Ladder. First off, I’ll admit that I’m a big fan of business fiction – especially when done well. There were no plot twists in the book, nor anything radical, but for me, it was a page turner (or button pusher in my case). The story documents progress (and setbacks) as a young exec meets with a business guru over the course of ten Monday morning mentoring sessions.

[1] Managing Humans: Biting and Humorous Tales of a Software Engineering Manager – is a fantastic book on management and IMO, essential reading for people managers.

DISCLAIMER: The above links are all amazon associate links (meaning that if you click the link and buy a book, Amazon credits me a few cents. I pull down the associate links automatically, but if you want to buy the books straight away, just go to Amazon directly and search for the above.

Inspiration for my blog posts often comes from whatever’s evoking emotion in my life (annoyances, victories, gripes, etc.). Three things happened today that ensured that I’d write this post. Here’s the story of a question from a seven year old, a phone call, and an email.

The first incident was a common one. My son asked me what the circumference of the earth was (ok – he didn’t use the word ‘circumference’, but that’s what he asked). Like a trained seal, I grabbed my smart-phone and looked it up. This was stupid, because a) I knew the answer, and b) even if I didn’t, I know enough about distances between locations and longitude to come up with a fairly accurate approximation.

Just a bit later in the morning, I had a short interview with a journalist from my university – apparently they’re hard up for stories and are reaching out to alumni for gossip. At one point, the journalist asked something like, “What advice do you have for students today?”. I gave a long answer that said (more or less), “A university isn’t a vocational school. Although you may get caught up in the rat race of learning facts and writing papers, to be successful in the ‘real world’, you need to know how to learn.” I also (once again) credited my Methods of Musical Research course and Dr. Snedeker for teaching me how to learn and for fueling my own passion for learning.

Less than thirty minutes later, I received an email response to a verbal request. I asked someone to research a topic and see if they could find any interesting papers that could foster additional discussion on the topic. They replied with a link to paper that shows up as the top link in Bing when searching on the exact phrase as the topic I presented. I expected my colleague to look at academic papers, and perhaps explore the references in those papers, or to find experts in the field and see what they wrote. I expected them to research…but I guess to them, ‘search’ was enough.

Perhaps I’m just being an old fart, but the serendipity of these three events launched a thought that scared the crap out of me. In these days of search engines and Wikipedia, I wonder if anyone knows how to think anymore. Knowledge is much more than learning or regurgitating facts. The learning that happens when you move knowledge from something you don’t know to something you do know is trivial compared to when you add discovering what you don’t know you don’t know to the top of the stack (1). I’m scared that too many people (not just testers) are happy with on-demand fact finding rather than true learning and discovery.

It could be just me – perhaps I spend so much time reading and trying to learn because that’s what works for me, and that on-demand learning is the true future. But if that’s true, I worry what kind of future that will be.

(1) For more on this theory of knowledge acquisition, read Philip Armour’s original paper on the topic here, or my takes, here, here, and here.

Yesterday, Michael Larsen posted a review of Weinberg’s book, The Secrets of Consulting. I’ve been meaning to share a few thoughts about this book for at least year, so I thank Michael for spurring me to finally share my thoughts and experiences with TSoC.

If you haven’t read this book, I bet it’s because the title scared you away. You’re probably thinking, “I’m not a consultant, this book is irrelevant”. For those thinking that, or for any others reading right now, I’ll tell you this. I have read (and in some cases studied), over 40 books on leadership – and this is one of the best I’ve ever read on the subject.

When people get to “Senior” roles at Microsoft, one of the things we expect is an ability to influence without authority, work through others, and basically make those around you better. For our introverted tech geeks, this is often a big wall to climb. Inevitably, when one of these Senior non—manager folks tracks me down and asks, “How can I get better at influence?”, or “How can I get others to accept my ideas?”, I hand them a copy of this book (I’ve done my best to pad Mr. Weinberg’s royalties with the dozen or so copies of this book I’ve purchased over the years). You see, being a good consultant, and being a good leader are pretty much the same thing. Some of my takeaways are below (adapted heavily to my usage over the years).

Good consultants know how to define the real problem at hand – they see what’s there, and what’s not there, and come up with a solution (or multiple solutions). I would hope, that if you’re a leader, you’d do the same.

Good consultants are humble – they don’t pretend to know it all, and know when to admit when they don’t. Your fee (or your title if you’re not a consultant) doesn’t mean you’re always right. It does mean, however, that you always need to help the team improve.

Establish and maintain trust. I can’t say enough about how much I value trust as a leadership attribute. The chapter on trust is critical for any leader.

Worry about the accomplishment, not the credit. This one rings home at MS, where our (ahem) review system often drives people to focus on credit rather than outcomes. The punch line, is that in my experience, the weight of the accomplishment always outweighs the weight of who did what. Just worry about making your projects successful, and the rewards will follow.

And given the hundred or so bullet points of wisdom sprinkled throughout the book, I’d say the meta-takeaway is that you have a good leadership toolbox. If something doesn’t work, doing it again louder won’t help – do something different. Use another leadership tool, or experiment. Nobody has time (or money) for a one-trick pony.

Now – to be fair, several points in the book are specifically for consulting – but if you read this book looking for leadership advice, that’s exactly what you’ll find (and you’ll barely notice the n/z sections).

What are you waiting for – go learn how to be a leader.

I try to read a lot about testing in blogs, articles, books etc. A few days ago, I came across this quote, and it struck me in an odd way.

“Commonly, test automation involves automating a manual process already in place that uses a formalized testing process”

The source doesn’t matter, as it turns out that sentence was copied directly from the Wikipedia article on Test Automation. I’ve been at Microsoft for a long time now, and although I do try to stay connected with testing outside of the Borg, sometimes I notice that *my* view of a subject is quite different than what’s “commonly” understood.

Or, maybe not – but let me try to explain my concerns here a bit more.

I’m all for saving time and money, but I have concerns with an automation approach based entirely (or largely) on automating a bunch of manual tests. Good test design considers manual and computer assisted testing as two different attributes – not sequential tasks. That concept is such an ingrained approach to me (and the testers I get to work with), that the idea of a write tests->run tests manually->automate those tests seems fundamentally broken. I know there are companies that have separate test automation team that do exactly this, and I think it’s a horrible approach.

Let’s use the Numberz Challenge as an example. I know right away that I’m going to perform some manual tests (e.g. make sure the app can launch and close (original version had a bug here), verify look and feel, color choices, etc.). These sorts of tests could be automated, but I generally don’t for a few specific reasons.

• The oracle for look and feel is difficult. Automated testing based on comparing screen colors or app dimensions is fragile. A pair of eyes for a few seconds every once in a while is much cheaper. There are, of course, exceptions, and if you’re convinced you can pull this off, just make sure you calculate the time spent investigating failures (and potential false positives) into your ROI calculation. I know many teams who have successful UI automation systems, but many more who have a bunch of test auto-crapation.
• Most systems I work on – and certainly everything that has a UI, gets at least a tiny bit of usage from the product team or a small pilot group before rolling out widely. If the screen is pink, or the font somehow changed to Comic Sans, it will be noticed without the need for an automated test.

When I look at something like Numberz, I also see where I need to use the power of a computer to answer some of the questions I have. For example, I know I need a large enough data set to be confident of the random algorithm, and I know I need extensive pattern matching to ensure that the next number is never predictable. Doing this manually is impossible (or a waste of time depending on what you try to do).

Now I imagine this scenario in the write-test-then-automate-it workflow.

   1: Test Case Steps:

   2: 1) Launch App

   3: 2) Press the Roll! Button 

   4:  

   5: Verify:

   6: 1) Ensure that none of the numbers is less than 0 or greater than 9

   7: 2) Ensure that the total field is correct

   8:  

   9: Repeat this test at least 10 times

Now, the “automator” comes along and writes this test:

   1: Loop 10 times

   2:   App.Launch (“numberz.exe”)

   3:   Button.Click(“Roll”)

   4:   // Validate numbers are within range 

   5:   // Validate correct total

   6: End Loop

From what little context I have from the interwebs, this appears to be a common scenario — but we’ve missed the critical aspects of testing this app (testing randomness / predictability). What we haven’t done at all in this situation is test design. To me, test design is far more holistic than thinking through a few sets of user tasks. You need to ask, “what’s really going on here?”, and “what do we really need to know?”. Automating a bunch of user tasks rarely answers those questions. (Note – it does answer some questions, so if you have a good system and tests you can trust, by all means don’t stop what you’re doing).

Where I think I have my big disconnect is in the definition of test automation. When I think of test automation, I don’t think of automating user tasks. I think, “How can I use the power of a computer to answer the testing questions I have?”, “How can I use the power of a computer to help me discover what I don’t know I don’t know?”, and “What scenarios do I need to investigate where using a computer is the only practical solution?”.

Perhaps test automation is purely the automation of manual tasks, and I’m attempting to overload the word. I know some folks prefer the term “computer assisted testing”, and I suppose that’s fine too.

To me (and I’m sure I’ve used this line before), it’s just testing. But please stop thinking of test automation as the step that follows test design, and start thinking of test design first.

For testers, being skeptical is generally a good thing. When someone says, “Our application doesn’t have any reliability errors”, I, for one, am skeptical. I’ll poke and prod and (usually) find something they haven’t thought about. There’s power in skepticism. Last year, I led a team of testers in performing code reviews of production code. My hypothesis was, that while developers perform code reviews thinking, “Does the code do what it’s supposed to do”, testers think, “In what conditions will the code not do what its supposed to do. You can insert the comment about testers being pessimistic (or overly pessimistic) here, but in general, the tester mindset is to question statements that seem…well, questionable.

But it’s easy to go overboard with skepticism. Time and time again, I hear good testers apply their skepticism broadly and liberally. Some (paraphrased) quotes I’ve heard recently include:

• “Model-based testing is interesting, but it doesn’t work in a lot of places”
• “I’m skeptical of static analysis tools – sometimes they have false positives”
• “Metrics are evil, because someone may use them to measure people”

I agree whole heartedly with each of these quotes. However, I worry that folks are throwing the baby out with the bathwater. Model-based testing (just an example), is a wonderful test design technique for stateful test problems. Although occasionally someone will screw up the value of MBT by claiming that it’s the only test approach you’re ever going to need, it’s just another technique (and the perfect technique given the proper context). Static analysis tools are also awesome, but aren’t perfect It’s good to measure some things too, but sure, one can screw it up.

I’m trying to think of anything in my life that works perfectly in every situation, but I’m coming up empty. I run into situations nearly every day where someone has a good idea that will obviously work most of the time – but not always. Given these situations, we could just send them back to the drawing board telling them, “I’m skeptical of your approach, because it won’t work in situation z”, but it’s probably a better idea to have a conversation about the limitations, understand where and when the approach may fail, and discuss mitigation or workarounds. Instead of throwing out the idea of running static analysis tools because of the potential false positives, discuss the false positive problem. Find out what causes them. Tweak the configuration. Do whatever you need to do to ensure the value of the approach.

Over the years, I’ve found value in some pretty stupid approaches. It seems that we should be able to find more value from some the ideas frequently discounted.

Even if we’re skeptical.

I’ve been thinking recently about what ‘code quality’ and what the phrase means. How does someone identify the difference between low quality code and high quality code (or medium quality code)?

In my quest for knowledge, I discovered a few interesting things.

I discovered the Spinellis book, Code Quality: The Open Source Perspective. I haven’t read this yet, but I ordered it, and looks like it will be interesting.

I found an article on coupling metrics on the IBM site. The reading here was good, as I discovered they have a series of articles on code quality.

I found a few references to analysis tools for JavaScript (some lint-like stuff)

Over in the high brow world of the linkedin groups, I saw a bunch of people referencing code coverage as an indicator of code quality.

Nothing earth shattering, but some good information, and enough to get me thinking about what *I* think code quality is – and what it is not. I think it may be impossible to define code quality in a way that everyone will agree with – but that doesn’t mean I shouldn’t try.

We have some metrics that can give us an idea about code quality. Quality code doesn’t have any compiler warnings (ideally at the highest warning level), and it’s clean from static analysis errors and warnings. Complexity metrics such as cyclomatic complexity (basically, the number of paths through a function), or dependency metrics such as fan-in and fan-out can also be indicators of code quality.

But – let’s make it clear that while I’m a (big) fan of those indicators, that’s all they are. Running clean with static analysis tools does nothing to ensure code quality – or product quality either (I ranted a bit about this a few months back). In fact, I guarantee you that I could write (or show you) a piece of software that compiles with the highest warning levels set and passes any static analysis tool with flying colors…that still sucks. 

One thing definitely unrelated to code quality is code coverage. All code coverage tells you is what percentage of your code you haven’t tested at all. If you squint and have a few drinks, you can pretend a little that high coverage rates from unit tests prove code quality…but once you sober up you’ll realize that all you have are unit tests that exercise some amount of your code.

But we can go beyond code analysis tools. Testability (can I observe and control program flow), and Maintainability (“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.” ref) are critical to code quality. Given a good reviewer, Thom Holwerda’s comic is also spot on – the only valid measurement of code quality is WTF’s per minute.

As I think about this, I find my mind wandering back to Pirsig. Quality and care are two different ways of looking at the same thing. Quality comes from care, so perhaps what I want to measure is the care put into coding. The fact that a developer wanted to compile at a higher warning level or turn on more static analysis checks, or that they cared enough to get the testers input on testability, or a peer’s advice on maintainability may be the best ways to measure code quality of all.

So now, perhaps, I’m in search of a way to discover and measure care.

Twenty-twelve has been a heck of a year so far. The day job has been keeping me swamped, but the work, the challenges, and the scope feel perfect – I’m having a great time, and I couldn’t ask for anything more in a job.

Historically, most of my blog posts come from experiences or inspiration from my day job (despite those who seem to think that every snide comment I make is about them, my snide comments are almost entirely about people I work with). While I’m sure that will continue to be the case, I’m still pretty deep in ramp up mode, and it turns out that the bulk of the work I need to do right now is solving a variety of test infrastructure and code issues. While I find this sort of work exciting to perform, I find it dreadfully boring to write about (apologies to all those who enjoy writing about these things).

But I do enjoy writing, and even as I write this trivial little blurb (a blurb that will turn into a blog post in about five minutes), I realize that I have a huge backlog of items not contrived from my day-to-day work that I can draw from in order to satisfy my need to write and share a few times a week.

Meanwhile, some chunks of my work are about things that I can’t blog about. In some ways, those are the most exciting. I have plans to share some of my thoughts on this when it’s appropriate, and I think it will be an interesting exercise (and hopefully, interesting reading when I can share). We’ll see what happens.

I’ve been on vacation (whatever that means) for almost a week now, and don’t plan to post again until 2012. To help fill in the break (and for anyone new to my blog), I thought I’d share some stats and popular posts from the last twelve months.

I wrote 60 blog entries in 2011. Some of the more popular posts (in no particular order) were:

• My job as a Tester
• What is Testing?
• Test Design for Automation
• Numberz Challenge
• Beyond Regression Tests
• R-E-S-P-E-C-T
• Judgment in Testing
• Lost in the weeds

I did a bit of speaking in 2011 (although, as usual, more than I planned). I was a pnsqc, at Intel in Israel, and at the Swiss and German Testing Day conferences. I also attended the second Writing About Testing conference (and had a great time).

Next year, I’m trying (again) to speak and travel less, blog about the same amount, and keep having fun working on the Xbox team. For everyone who’s read or posted a comment, thank you for supporting my silly addiction to writing. I hope I’ve given you something to think (or complain) about.

Best wishes for the holidays and 2012.

I’m a Senior Tester, don’t you know

Been testing software for five years or so

I got bills to pay and need a better job…

Looking for jobs with ‘Senior’ in the name

Gotta’s admit, they’re all pretty lame

Looks like I’m over-qualified…

Someone (non-Microsoft) asked me recently what a “Senior Tester” does. I told them that I knew in the context of Microsoft, but really had no idea if the term meant anything in the industry. After a quick (and painful) trip to a jobs website, I was able to confirm that the ‘Senior’ title doesn’t carry a lot of weight. Consider some of these quotes – all from positions advertised as “Senior Tester”, or “Senior QA Tester (on a side note, wtf is a QA Tester?):

• With minimal direction writes scripts and executes test cases during functional/system/integration testing.
• Modify existing test plans and testing information to correct errors, to adapt to new test scenarios, upgraded interfaces and performance enhancements.
• Monitor bug resolution efforts and track successes.
• Participates in analysis and design walkthroughs, as well as team meetings
• The Senior Tester will work extensively with the Test Lead to provide test estimates
• Complete knowledge of QA Best Practices and the full Testing SDLC

Let me rant on the last bullet point first. Actually, never mind – anything I can say is something you’ve probably already thought.

As for the first five bullet points, are those really considered “Senior” activities? I was surprised that being able to cross the street by themselves or washing their hands after using the bathroom weren’t required attributes. Then again, I’ve been sheltered in the world of Microsoft for the past 17 years (and as a noob in a startup for 2 years before that), so my world view of testing has limitations.

So, let me talk about what I do know. Here at the evil empire, we have standard titles for all roles (more details on test titles are here). As I mentioned in the link (which I know you haven’t read yet), we have a “Senior SDET” title at MS, and this is where we expect testers to exhibit some leadership skills (they learn how to find the bathroom by themselves at a much lower level). To be clear, leadership does not mean management. ‘Senior’ is the minimum level where we like people (who are interested) to move into management roles, but we have hundreds of non-managers at the Senior level (about 8% of the total test population if you’re curious).

My random search for Senior test roles wasn’t entirely in vain. Consider the following:

• Experience with leading testing efforts on integrated teams and developing and implementing sound testing methodologies
• This is a senior role that includes evaluating existing automated testing process and recommending improvements

Now those seem a little better – now the role requires some leadership and decision making, so let’s look a bit closer at these a few other attributes I expect to see in Senior testers at MS.

• Decision Making – senior testers make confident decisions, and do not rely on consensus for making (most) decisions.
• Influence – Dwight Eisenhower said, “Leadership is the art of getting someone else to do something you want done because he wants to do it.”  Senior testers make their team better through influence rather than mandates – they are the point guard of the test team.
• Network – senior testers use their network to find answers (or knowledge), and connect their teammates with their network (i.e. tester matchmaking)
• Credibility – senior testers don’t proclaim they’re a leader, and no title demands respect. Leaders know that respect as a leader comes through establishing credibility with their team and peers. For me, the moment you demand respect as a leader is the moment you lose my respect as a leader.

That’s not an exhaustive list, and each of those points could be (and may be) a post unto its own. But that’s what ‘Senior Tester’ means to me (In my sheltered life). Undoubtedly, your definition is different, and that’s fine. Regardless of where your opinions lie, I hope this provides some food for thought on your own views of what ‘Senior’ means.

A little over five weeks ago, I got a new job (one of the things I find great about MS is that one can change jobs completely without going through the hassle of filling out new forms). In that five weeks, I was in Germany for four days, and off for two days for Thanksgiving, so technically today is my (…counting on calendar…) 23rd day on the Xbox team.

Admittedly, I’m still in the honey-moon phase, but I’m having a great time. But life just doesn’t seem to be as fun without a surprise here and there, and this career move wasn’t without exception. More on that later – but let’s first consider a bit of career advice.

One tidbit of career advice I often give is “The 3 Ps”. When you’re looking for a job (or evaluating your current position), you need to consider the Product, the People, and the Person.

• Consider the Product – You’ll be more motivated and excited about your work if you have interest in the product (or the technical challenges regarding that product). On the corporate applications side of Microsoft, I honestly think that Lync (my old group) is freakin’ cool, but there’s no understating the sexy coolness of Xbox and Kinect either.
• Consider the People – Nobody wants to work with a bunch of jerks. I don’t want to work with people who don’t care about their jobs. I happened to know dozens of people working on XBox (I worked with many of them on Windows 95 & Windows 98), and all are great people. When I was considering the move, I chatted with people on the team that I didn’t know – and they all had this vibe of loving what they do. To me, that’s the sign of a good team to work on.
• Consider the Person – We all have to work for someone. Take time when considering a new position to make sure that whoever you report to (and ideally, their manager) are on the same page as you; and will support you and help you grow.

To me – the Person part of the equation is critical. I’m in a rare position at Microsoft – we don’t have many testers at my level who do not manage teams. Managing someone like me, and communicating my value (or lack of value) appropriately is a challenge. Because there are few of us extra-senior testers, there are very few managers who have experience using us effectively. I selected my last manager based on this criteria, and when I moved to Xbox, I used the same criteria. It was hard to leave a team where I had the 3 Ps, but the time was right, and I took the time to ensure that I would have the 3 Ps on my new team as well.

Three days into my new job, my manager called me into his office and said, “Welcome to the team – we’re having a re-org”. The details of the re-org aren’t important, but they make perfect business sense, and I’m excited about the change. Unfortunately, the manager I had carefully selected wasn’t going to be my manager any more – and with the upcoming release of the new dashboard, I wasn’t sure where I was going to land until a day or two ago.

So – with one manager on the way out, and another yet to be defined, I fell back on what I seem to do best – I did whatever I wanted. In the land of Xbox, I suppose that could mean a whole lot of Skyrim and Forza, but I dug deep into some testing challenges, looked at engineering systems, got to know people and basically spent a whole bunch of time figuring out how the org and technology work. I once considered a position at a company full of young folks where someone suggested that I spend at least a month or two “doing whatever I wanted” until I figured out what I really wanted to do. My recent experience wasn’t quite as directed as that, but I find it interesting that concept seems to repeat throughout my career.

The good news is that I’ve met with my new boss a few times, and I think we’ll work very well together. We’re still figuring out exactly what I’ll do (which is perfectly ok), but I’m completely comfortable with how he wants to use me, and although he’s still a bit of an unknown to me, I’m pretty confident my 3 Ps are still in place. I’m excited, and a little scared (another bit of good career advice), so I know I’m in the right place.