The New World

I mentioned on twitter that Barack Obama and I both left our old jobs on the same day. The world has been a very different place for both of us since then.

Twitter is full of politics – and I’m completely ok with that, and happy to join in with my own opinions and thoughts. I’ve marched, I’ve protested, and I’ve done a lot of learning (and re-learning) about civics, politics, and done what I can to become as educated as possible.

Meanwhile, I have a new job, and a lot to learn about a new company, code, practices, people, and culture. I absolutely love it so far, and I know I’m only scratching the surface of what there is to learn. I’ve asked a lot of questions (even a few that actually seemed smart), but I’m still at the stage where everything I discover unveils three more things I need to learn.

In eight days at the company, I’ve spent two days in San Francisco (where we’re hiring testers for my team in Ads and Analytics – ping me for more info), and two days in San Diego (where I crashed a hack week to meet more people from my team). Combine that with the work-at-home-snow-day that was Monday, and tomorrow will be just my fourth day at my desk (but I think I’m in a non-traveling state for at least another week).

My new team is spread across offices in Bellevue, Austin, Helsinki, Odessa, Copenhagen, and Montreal. I’ve been to half of those cities before, and look forward to visiting folks in the others (odds are that Helsinki is up next).

Too early to share Unity testing stories, but I’m sure those (or stories inspired by Unity) will be on the blog soon. Until then, wish me luck on trying to get a reasonable drink of water from the current firehose of learning.


For those of you who missed it on Twitter or in the AB Testing Podcast, my unemployment lasted (as planned) just a bit over week. On January 30, I started work at Unity, heading up quality for their various services.


I spent most of last week trying to learn as much as I can and trying to meet  and get to know as many people as I can. I expect the next few weeks will be similar.

I’m massively excited to join Unity, and know I’m going to learn a lot. It’s different than the big M, but so refreshing in so many ways.

More updates here, as they happen.



The Breakup

This one goes out to the one I love. This one goes out to the one I left behind

Relationships are both challenging and rewarding. As adults, we’ve all gone through dozens of relationships – some short, some long, some very long. They all have their ups and downs; their ebbs and flows; and their joy and pain. This is an open letter reflecting on the longest relationship I’ve ever had.

It’s been over 20 years (22, actually) since we first crossed paths. In the early days, I was so excited to be with you. I knew of you, of course, long before our paths crossed –  but couldn’t believe we were together – you were completely out of my league. There was no reason someone like you and someone like should have been together. But it worked – for both of us. I think it worked better than either of us thought it would. For many reasons you brought out the best in me. In some ways, you still do.

We both grew a lot in those early years. I gave you everything I had, and you were there to support me and grow with me. I was proud and excited to be your partner.

As with most relationships, there were ups and downs. I was mad at you sometimes – other times even livid. Sometimes I didn’t respect your decisions – sometimes I don’t feel like you respected mine – but as we grew older together, we both began to change and grow – but I guess that’s normal in any relationship.

The problem is, that we’ve been growing in different directions for many years now. I give you full credit for the effort you’ve put into changing and growing these past few years, but for me, it’s just not fast enough. I know you want to change, but I feel like you just have too much baggage from your history to ever change enough where I can be happy with you. I want to race through life like a cheetah – and while you’re no longer the tortoise you were for much of our relationship, you’re at best, still just a bear lumbering through the forest.

I need more.

What’s happening now isn’t your fault – and it’s not mine either. We  grew in different directions, and now it’s just too much for us to recover. At least it’s too much for me.

I’ve been thinking of leaving for at least five years. It’s been easy to stay – you’re comfortable, and I feel safe with you.

But I’m not happy. I need more – I need to be challenged more; I need to grow more; I need to be around those who won’t be afraid to take risks and fail with me as I learn.

I took some time over the last year or so to try even harder to make-it-work. In the end, I just don’t feel like I’m getting the same from you of what what I put into this relationship. I don’t feel like you truly care about me and the value the things I bring to the table.

It’s just not working.

Could I have tried harder – of course. Would trying differently have made a difference – likely… but in the end, there are just too many incompatibilities for me to get over. It’s sad and scary, but I need to move on. I need to give myself a chance to have a life without you and see if I’m up for the challenge.

I’ll miss you. I appreciate everything you’ve done for me, but our time is over. It’s time for me to move on and see what I can do without you.

My last day at Microsoft – the company where I’ve spent the last 22 years, is Friday, January 20.


Asking Questions

I love reading twitter after I give a talk, because I can review what I said and make sure I agree with it (I never know what will come out of my mouth when I’m on a stage in front of people – it’s part of my presentation). At TestBash in Philadelphia, I (apparently) said:


I sad the above at the end of a story where I also referenced Richard Feynman’s story about being afraid to ask a question, asking anyway, and finding a “bug” (in this case, a problem resulting from a stuck valve). If you’re a tester, and haven’t read Feynman, you are really missing out on the stories of a great systems thinker. In this story, Feynman asks a question he is almost too afraid to ask, and the answer is, “… ‘you’re absolutely right, sir.”

In my day job, I have a shallow view of a big and complex product. I review some code, I look at some of the specifications and designs, and I go to some architecture meetings. I know enough to know that I don’t know most things, so I ask questions when I’m confused.

Inevitably (and most recently, just a few weeks ago), I’ll ask a question that makes me look smart despite my lack of knowledge. In this case, I happened to look at a code review, and was curious how the feature and flow worked. I reviewed every line, but it still wasn’t clear how one part of the flow worked. I dug for a while, but all I did was feel dumb about not being able to figure it out. As a last resort, I added a comment to the code saying, “It’s not clear to me exactly how [this flow] works, do you mind swinging by my desk and walking me through it so I understand it better?”

The response in the code review? ‘’”you’re absolutely right – great catch!”

I wouldn’t bother telling the story if this was a one time thing, but it happens all the time. I try to learn, and ask questions when my learning is (or feels) blocked. Most of the time, I merely learn, but sometimes, I inadvertently find something interesting. Either way, if you’re passionate about learning (and every knowledge worker should be), then always ask questions if you need clarification or more understanding – you may just be right.

Watch out for the HiPPO

Way back in 2008 when I wrote chapter three of HWTSAM, I briefly mentioned the HiPPO in the context of ship room (war room) meeting structure:

Everyone’s voice is important. A phrase heard in many war rooms is “Don’t listen to the HiPPO”—where HiPPO is an acronym for highest-paid person’s opinion.

It was a term I came across, but could not find attribution. I discovered fairly recently that Ronny Kohavi (whom I’ve stolen  / borrowed from liberally when discussing A/B testing and experimentation) who came up with the term in 2006.

My old colleague Seth Eliot also discussed the HiPPO in a blog post in a post last year.

While many organizational leaders aspire to become the next Steve Jobs, just about all of them will fail (as an aside, I was in a meeting once where a very senior person said that they made a critical decision, because “it’s what Steve Jobs would do, , and I’m like Steve Jobs”). What I’ve discovered is that no matter how strongly someone feels they “know what’s best for the customer”, without data, they’re almost always wrong.

It can be a big step to use data over organizational rank to make decisions, but if you care about pleasing your customers, the choice is obvious.


I mentioned this at TestBash a few weeks ago (and on a yet-unreleased ABTesting podcast) – and probably on twitter too, but my third “secret” project in a row at Microsoft (following Xbox One and the science project to make Android apps run on Windows Phone) was finally announced to the world on November 2nd.

For the last fifteen months and change, I’ve been in charge of quality for Microsoft Teams. I was originally hired as “the quality guy”, where my job was to coach developers on testing, and ensure we had a strategy in place to deliver a quality product. Along the way, I picked up some infrastructure pieces and now am responsible for taking code from developer check-in all the way through do deployment. I’ve found that owning check-in gates, builds, and deployments (along with quality tools and testing) gives me a lot more confidence in releasing a high quality product to our customers.

It’s also been a great opportunity to catch up on a lot of tech and processes that I’ve wanted to get involved in. We use a lot of open source, and we try to release as frequently as we can (currently weekly, but certainly on the path to continuous deployment). There’s still a lot of work to do on the product, and a lot on my team to ensure we can improve builds, deployment, and quality.

Now that we’re “out there”, I can finally share some stories about things I’ve tried, what’s worked, and what has failed using real examples rather than abstract references. Looking forward to sharing those stories soon.

TestBash Smash

Short story is that TestBash may be my new favorite testing conference. Great venue (no, fantastic venue), well-organized, and excellent variety and diversity across the different presentations. I was reflecting while walking the streets of Philadelphia this morning and realized that almost every talk was experiential – filled with stories of problem solving and discovery.

For those who know me, let me put it this way. I don’t think I rolled my eyes even once! (except the time when Nancy asked how many of us were context-driven testers).

I also met so many people I’ve only known on twitter (so many, in fact, that I can’t list them), and a large handful of new people I follow now too.

As for my talk, it seemed well received for a talk that (in title) could be scary to a lot of testers. Honestly, I don’t think the talk would have worked as well with a less sophisticated group of testers. It’s really easy to take an idea like testing without testers and look at it with no critical thinking and dismiss it entirely – but folks seemed to get how it could work (and I, of course, was unafraid to talk about how it can fail miserably.

I used a ping pong metaphor to describe the inefficiencies in typical test-dev relationships, and I hammed it up a bit figuring if I was going to look stupid, I’d go all in. It went a little  better than I expected, and I owe Angie Jones an apology:

To my credit, Angie – I don’t have an A game. I barely have game, so I just wing it and see what happens. This time, ping pong worked – I was fortunate and lucky.

Overall, a very good experience, and a conference I look forward to attending again someday.

Roles and Boxes

The fine folks at the Ministry of Testing keep promoting my blog posts, so the least I can do is give them a link and a shout out. I’m looking forward to talking about “Testing without Testers” at Test Bash Philadelphia (preview here) and about my role on the team.

This morning, I passively listened to The Bach Bros talk in a webinar about roles and what they mean, as I was curious to get their take. The punch line of the talk was the introduction of “Role grams” – which are shapes that describe who does what, what they’re doing, and who owns it. Nothing earth-shattering, but a workable model.

For the last 14 months, I’ve been a manager on an engineering team responsible for deployment and quality. As I’ve put it before, I am responsible for everything that happens between the time code is checked in (or slightly before, as I also added a few git hooks) to the time it is deployed to our production servers. Given that context, my role is a big blob of stuff. One could say that my “role” is Director of quality and infrastructure, and within that role, I take on activities of tools, engineering productivity, builds, quality, and testing. Certainly, you could say that each of those are roles I take on as part of my job – that model just doesn’t work as well for me. My role – the part I play as an actor on my product team, is one where I need to look at the entire ecosystem of check-in, CI, build, test, and release as one thing. I look at all of that as a system – how quickly and efficiently can I move a developer check-in to something that shows customer value. I have to view the system to know which parts are bottlenecks, and which parts need speeding up, or slowing down in order to increase overall system efficiency.

Here’s a low-tech view of my role. 20160915_120808I’m not locked in my box – in fact, since I’m the “quality guy” (label, not role) on my team, I spend most of my time working across the team on improving the testing done by other engineers on the team. I also have a small team of dedicated testers (these people have a testing “role”), who focus almost entirely on exploratory testing driven by trends in customer feedback and areas where I think we need additional breadth coverage in order to reduce risk.

It’s an evolving role, but one I enjoy, and one that I think will become much more common in the coming years.

The Mojo of the Weasel

It’s been a heck of a year.

I joined a new team at MS fourteen months ago, and it’s been the busiest fourteen months of my career. To be fair, I worked more hours during my stint in Xbox One, but the role I’m in now has even more responsibility (more on what that means some other time). Since I haven’t really blogged much since I joined the team, perhaps a recap is in order for context.

I’m working on a yet-unannounced, and possibly-soon-to-be-released product. I was hired as “the quality guy”, but expanded the role over the time I’ve been on the team to take ownership of all of our build and release infrastructure as well. Basically, I’m responsible for everything from the moment code is checked in (including check-in quality gates), until it hits our production servers (at this time, “production” is for beta users only). This includes our CI system, build systems, test infrastructure, deployment, and a bit of manual testing as well. To this end, I’ve ventured back into management, and manage a small handful of full time employees, as well as a larger handful of temporary (vendor) testers. I’m responsible for making sure we have systems and strategies that enable us to get a good product to our customers frequently. I make sure the code is ready, and that the product is well-tested. It’s also my call on whether – and when to push bits to our production environment. It’s a killer job, but one deep in my wheelhouse. I’ve grown in many ways over the last year, and learned even more.

But it’s taken away a bit of who I am. I don’t write much anymore. I speak much less than I used to, and I’ve all but disappeared from twitter (one plus is that I’ve been really happy with the AB podcasts that Brent and I have been delivering). While part of me is growing and learning, another part of me is withering and dying.

So I need to make a change. Not necessarily a change as big as changing jobs or changing companies, but I need to remember that my job is just a job, and it’s not who I am. Like a lot of people, I lose sight of that sometimes, and I need to see if I can get myself back on track – and start by (re) connecting with the community that inspires me and drives me to do even better things.

I remain excited about the product I’m working on, and the team I work with – but perhaps more excited about speaking at Test Bash, getting back to talking with old friends on Twitter, knocking the dust off, and sharing a bit more with my friends.

Here’s to re-engaging.

The A Word returns (sort of)

Chris McMahon, who has always impressed me with his words and his wit called me out in his blog.


Apropos of my criticism of “Context Driven Approach to Automation in Testing” (I reviewed version 1.04), I ask you to join me in condemning publicly both the tone and the substance of that paper.

Almost exactly a year ago, I reviewed a draft of the paper, and my name is among those listed as reviewing the paper. The feedback I gave was largely editorial (typos and flow), with a few comments about approach that I’ll repeat here:

The first red flag I called out (and will call out again here) is the phrase that describes test automation as something to “automate testing by automating the user”. This is a shallow view of test automation, but other than comment, I didn’t push hard on it. In hindsight, this was a mistake (much of The A Word touches on this topic).

In regards to the scenario the authors chose for scenario automation, I thought the choice was weird, and asked for more…context and provided some food for thought.

I think you can add even more emphasis to how and why you chose to automate scene creation. Too many times, testers choose to automate because they can automate. The thought process (for me) may be, “The scene feature is pretty important, I’m curious what happens if an author has thousands of scenes. Will it cause performance problems, formatting problems, file load problems, or other issues, etc. There’s no way in hell I want to do this manually, so let me write some code to help me run my experiment”.  You may also want to discuss alternate implementation ideas – e.g. creating a macro in Notepad++ to create the text then paste it in, or creating a macro in Word for the same, or using for in a windows console (e.g. for /L %f in (1,1,1000) do echo “##”>>output.txt&echo “Scene %f”>>output.txt ). Using tools for creating and manipulating data could be a whole article.

And that led to my real beef with the paper. It talks about using tools to test – which can be a good thing, but it doesn’t really talk about automation in the way successful teams actually use it.

I think it may be important to talk about purposes of automation and where to apply it – or at least one context of that – as I don’t think that’s discussed enough (but I’ve made a note to myself to write a blog on this very subject). At a BFC (big company) like Microsoft, we write a lot of shared / distributed automation – automated tests that we need to run on a lot of different hardware/platform configurations in some sort of lab setting (tools like SauceLabs or BrowserStack are helpful here). Web apps are famous for this [problem].

I also commented:

The other kind of automation (which you cover in your article) is what I sometimes call exploratory automation (or more often, just “Testing”). This is where we get a test idea, and want to write some quick automation to help learn about the product. While we may turn this sort of test into something that’s distributed / shared someday, its primary purpose is to help me answer questions and learn. There’s (another) story in HWTSAM where I described a case of this. I wrote really ugly brute force automation in C (using things like FindWindow and SendMessage(LBUTTON_DOWN…) to simulate opening and closing a connection to a remote host many times (the only thing this app did). It found a nice memory leak that may not have been found otherwise (or at least not as quickly).

All of this feedback fed my uber-point, which was that while the article talked about test automation, the examples really just talked about using somewhat random tools to help the authors explore or test some software. There was nothing about strategy, or about more typical use of test automation. I asked about it in a comment:

I wonder why you don’t use the word “Tools” in the title – e.g. “A CDT approach to tools and automation in testing” or something like that.

…because the paper is about, as I said above, using non-standard tools to help test. Sure, it’s automation in a sense, but nothing in the paper reflects the way test automation is used successfully in thousands of successful products.

All that said, I do not support this paper as a description of good test automation, and I think it’s an inappropriate method for anyone to learn about how to write automation. Chris requested that the authors remove the paper; while I support this, and do believe that the paper can cause more harm than good, there’s so much bad advice on the internet about creating software, that removing this one piece of bad advice will hardly make a dent.

I did not realize my name was listed as a reviewer, and although I did (as admitted above) review this paper, I do not want my name associated with it, and will request that the authors remove my name.

%d bloggers like this: