We get feedback on the courses we teach for testers at Microsoft. My team is pretty good at reviewing feedback and making adjustments as necessary. I remind my team of the same thing. They need to treat the positive comments (“the instructor made the class awesome!”) and the negative comments (“the instructor was an idiot”) the same. Either throw them both out, or treat them both as valid. Obviously, you can play the balance game. If 20 students said “great”, and 4 said “sucks”, then you’re probably on the right track (although you’d want to self-evaluate on why the polarity exists in the first place). It’s a frequent topic of discussion, but one that I think everyone understands.

The concept came up again this week when a teammate pointed out that someone commented about me on the mini-microsoft blog. The comment in question was mostly positive, yet it was in response to something negative (which, in full disclosure, I replied to a week or so ago). In this case, I either have stale ideas and have some talent, or neither. I don’t really care either way, but I suppose I’m happy for the attention (there’s no such thing as bad publicity).

The important lesson to learn is that there’s no bad feedback, and no irrelevant feedback. There’s just feedback – it’s your choice to listen to it or not.

I expect that this year, among other things, will mark the return of the Weasel – stay tuned.

(part 2 is here)

(part 3 is here)

(part 4 is here)

I’ve been in the Engineering Excellence (EE) team at Microsoft for nearly five years now, and the time has flown by. The job is completely different from a typical test role, yet almost all I talk about is testing. I’ve used my time in this group to continue to learn about testing, but to this day, I still miss the ebb and flow of adrenaline that comes from working on a shipping product.

The first big project assigned to me in EE was to design a course for our senior testers. I had no idea what I was doing, but I after talking to dozens of people around the company and receiving some critical help from some of my new peers, I ended up with a course that has been well received for going on five years now, and is relatively unchanged from the original. This work led to a ton of research over the last five years on growth paths for testers (one output of that research is in my pnsqc paper from 2009). I think of the work I’ve done in EE, this work on growth and career paths for testers is what I’m most proud of.

About a year and a half after joining EE, my manager left to join another team at Microsoft. Less than another year later my “new” manager left and I, for some reason, decided to step into the director role on the EE team. I’ve never been completely comfortable in a manager role, but after some long talks with my peers (soon to be employees) about the move, I decided to give it a shot. The core of my work didn’t change much, but I did spend more time on “overhead”  – budgets, scheduling, etc. I’ve had a chance to learn a lot in this role, and a chance to work with test leaders all across the company.

I’ve also used my time in EE to study. I’ve probably read a hundred books about testing, management, and leadership; taught dozens of classes and given more internal talks than I can count. I’ve spoken externally as much as I can handle, and wrote more about testing than I ever imagined I could. All in all, I have a lot to be proud of about the last few years of my career.

What will the future bring for me?

I’ll tell you later.

(part 2 is here)

(part 3 is here)

When I joined the CE team in 2000, l knew nothing about embedded systems, and exactly as much about non-Intel processors. Furthermore, I didn’t know what I was supposed to do on the team. The Pocket PC had just come out, but the team was already pretty much set for the next release. Given my recent kernel experience with Windows Millennium, it made some sense for me to work on something similar on CE. I was still in the reluctant test lead role at this time, so they assigned the kernel and file system testers to work for me. The problem was, that they had it covered so well that there wasn’t much left for me to do. They gave me some other stuff to own (like drivers), but they had it covered to. Sure, I got to help out, but I was sort of turning into one of those managers who does nothing but manage.

But I still had some time, and I was looking for something “technical”. One day in a meeting, I discovered that they were going to punt/postpone every single bug and feature in the shell code (shell is the GUI on top of the OS – the basic explorer shell on windows). I volunteered to do the work and for some reason my manager and hid counterpart on the development team agreed. My role became the Kernel/Filesystem/Driver Test Lead / Shell Developer. I can’t recall how I actually survived this dual role.

Working on the shell was fun, but more importantly, it was a turning point in my career. When I agreed to take on the work, I thought it was my stepping stone to a software development position. I worked closely with the dev manager and hit all my deadlines – but I also discovered something important about myself. The workload was fairly light. I was supposed to fix 40 or so bugs and add 4 features. I love fixing bugs (actually, I love the investigation process). I love to figure out why things don’t work. The features on the other hand, were sort of ho-hum. The only thing I remember writing from that time was the feature that calculates the daylight savings time rollover. I’m not sure if my code is still running in windows mobile, but at one time it was. When I finally finished all of the work and had some time to reflect I realized something important.- I didn’t really like being a developer. I was certainly capable, but the core of the work didn’t align with the core of my passions. As soon as I finished my features and fixed my bugs I met with my manager and I told him that I wanted to be a full-time tester; no more development work, and no more management. This was probably the first time in my career I really took testing seriously.

Not too much later, I became a test architect. That title doesn’t really denote a specific role at Microsoft – at the time it was more of a fancy title they gave testers in senior roles. I started meeting with other test architects (we had, and still have, a weekly meeting). There were 6 or so of us at the time and we had a handful of projects we’d work on together. We gave a presentation to Bill Gates in December of 2000 (I remember, because the presentation was on my birthday), and it seemed to go well. At the same time, I began to study testing. I started reading books about testing. I found contradictions, good ideas, bad ideas, and things that made me think of new ideas. I know a tester who says that there are no good books on testing, but I disagree completely. Every book on testing is good as long as you read it critically and use the knowledge in the book to confirm or challenge your own ideas.

So, I studied, I read, I tested. I was having fun experimenting and trying new ideas out. I began blogging while on the CE team – mostly writing about the testing tools we used for Windows CE and sharing some of the utilities and slides I was showing at developers conferences. It was a time of a lot of growth for me, but by 2005, I felt I was ready for something new. I had been in CE for about 5 years, and we were wrapping up CE 5.0. One of the testers I had mentored for several years had the passion and knowledge to be a test architect, and it made sense for me to step out of the way. I wasn’t sure where to go, so I asked Harry Robinson if he had any ideas. Harry was in Engineering Excellence at the time, and I thought he may have some insight into what groups could use someone like me. As it turns out, Harry told me that he was leaving Microsoft, and that I should apply for his open position. So – I did, and in April of 2005, I transferred to the Engineering Excellence team.

(part 2 is here)

When I joined the Windows Millennium (I usually cover my mouth and mumble when I mention this product) team I was given the opportunity to join either the development or test team. I met with both the dev and test managers and disliked the test manager much less :}, so my mind was set.I also enjoyed testing a lot – I liked figuring out how things worked (and figuring out how they didn’t work). One thing that bugged me during Win98 development was that we stopped testing on the debug version of windows. The debug build caught all kinds of errors that may not be caught until much later (or ever) (sort of a fail-fast mode) – it was a great way to make sure applications, libraries and drivers were all written and behaving well. Unfortunately, it suffered from broken windows – by the time the winme team was formed, it wouldn’t even boot without going through dozens of breakpoints in the debugger. When you got that far, a bunch of the other windows apps would hit breakpoints. The big problem was that the app breaks were mostly benign (e.g. bad NULL parameters), but because it was such a pain to run, nobody could take advantage of all of the other checks built into the debug build. The best part was that when I said I wanted to fix it, our management team told me to go for it. The initial job was just a lot of error / parameter fixing. It only took a few days until the debug OS would boot, then I fixed all of the apps I could. At this point I had to work on the “play nice with others” part of my job – I had to get a bunch of teams outside of windows to fix their application bugs. Eventually, I got most of them fixed, and you could actually “self-host” (use the os for most daily work) the debug build. We even used to play network games on the debug build on Friday afternoons! We did a lot of enhancements to the debug build – memory leaks and memory corruption were pretty easy to track down and fix, and we found a ton of other pretty bad errors (just not all of the errors customers found)

Along with debug windows, I also was asked to “maintain” the win9x debugging tools. The kernel debugger ran as a TSR that loaded at boot time. It communicated over a serial cable (although we later made it work over 1394) to a smart terminal. It turns out that “maintenance” of the debugger also included a bunch of changes and enhancements. One bit of context is that the winme team was “only” about 200 people (possibly less), but at least a thousand people around the company tested and dogfooded winme, and most of those used the kernel debugger. This meant that if I messed anything big up, that I could have a thousand people blocked. Luckily, that never happened – I made it through the product cycle relatively unscathed.

Along the way my manager tricked me into becoming a lead. He gave me more work than I could handle and when I complained he gave me “the new guy starting next week”. Turns out that guy was pretty smart and helped me look good for quite a while (we still have lunch every few months to this day). I ended up with 4 or 5 direct reports – one helping me with debug windows and the debugger, and the others working on the overnight stress infrastructure. I spent most of the day every day in the debugger and learned a lot. I’d use it to explore the operating system – setting breakpoints on unfamiliar functions so I could step through and see how they worked. I would change values (including return values) and watch what happened in untested code paths. I found a few cool bugs this way, but mostly learned.

Just before WinME shipped, our GM was promoted to VP. He was going to take over the Windows CE team and in the summer of 2000, he took 20 or so of us with him. In just over 5 years at Microsoft, I had worked on win95, win97 (the product I was working on before PST split into windows and IE teams), IE2, IE3, win98, win2k and winme (and 3 blog posts). The good and bad news is that I can probably finish the next 10 in one more post.

My official full-time start date at Microsoft was June 6, 1995. The first release candidate for Windows 95 was June 8, so in a way I started just in time – I started in time to get the Win95 ship gift (limited edition of windows 95), and also got a watch commemorating 20 years of Microsoft. I think we released the US version sometime around the 24th of June, and as a full time employee, I was technically allowed to attend the various ship festivities. I remember a gathering behind building 4 where way too much Dom Perignon was poured on people’s heads, as well as a larger party at the Shriners club in Seattle. I attended both, but I sort of had to sneak to the first one. I had a lot of responsibilities for the Asian versions of Win95, and I was supposed to be making sure that nobody in Japan ever had anything to complain about when running windows 95. But, I figured that since I gave up my overtime to get a blue badge, that it was ok if I went to check out the parties (note that I did the same thing for the official August 24 launch party). Windows 7 shipped a billion or so languages on the initial release day, but back then, the goal was to ship all of the top languages within 90 days (don’t ask me what “top” meant – what it meant to me was that i was working on Japanese, Chinese and Korean Win95 until the end of August). Eventually, we did ship, and it was time for me to be part of the early product cycle for the first time ever at Microsoft.

So, for a few weeks in September (I think) of 1995, I wrote some completely kick-ass tests that eliminated some of our brain-dead manual testing (I know I need to watch my words – not all manual testing is brain dead, but the testing I replaces was). I wrote a controller that applied configurations to multiple machines (think Japanese computer and share names), then ran a suite of network testing against those specific names to verify there were no problems with any Asian ideographs and network connectivity. I showed my (still awful) boss, and he almost smiled. He was so excited, in fact, that he set up a demo with his boss for the following Friday. I spent the weekend tweaking and polishing my suite to make sure it would survive the demo like a champion.

But the demo never happened.

Instead, on the day before the demo all of “PSD” (we were called the Personal Systems Division at the time) were hauled off to a nearby conference center (the MS conf center wasn’t built yet) so Bill could talk to us. He talked about his Internet Tidal Wave memo and said some other stuff that I’m sure was important. All I know is that by the time I got back to my office, I was on the Internet Explorer team.

I worked a bit on IE 2 (although, to this day, I can’t remember what I tested). IE3 was the “Netscape killer” release, and was on a 6 month schedule (Feb-Aug, 1996). I owned the feature that let you select the language of the page. At the time, languages weren’t auto-detected – instead, you clicked a little globe in the status bar and selected the language you wanted the page to display in. My testing consisted mostly of creating a bunch of web content and putting foreign characters in every html element I could find to ensure that they displayed correctly. I also remember that we changed our behavior with named entities. Before IE3, for example, a Russian page could display the character “?” by using the two named entities. One was the character code point – e.g. &196;. The other, unfortunately (blame front page), was by using the named entity for the code point in English – i.e. Ä (“A” with an umlaut). This, of course, is stupid,  but the change was going to break hundreds (the 2010 equivalent of hundreds of thousands) of web pages. I remember a lot of panic and worry among some of the managers. I also remember suggesting that I write a tool to help web masters fix their broken html. I wrote the tool in less than a day and received heaps of praise. Bing tells me that there’s still one web page out there about this tool – kind of funny (and embarrassing, since I’m sure it’s the tool is nothing I’d be proud of today). I also wrote some IIS server extensions that had something to do with language detection. It was interesting – and fun, but I knew that web browsers weren’t my calling.

A few weeks after IE3 shipped, I re-joined many of my friends from the previous year and joined the windows 98 team. The first project I worked on was a suite of networking tests – this time written in C and tested at the API level. I felt like a real programmer for the first time and got a lot of mileage from that little test suite. It was flexible enough to test the majority of network functionality against a variety of servers (anyone remember Banyan Vines?). I was on the international team, and as one of the few people on that team who could program well, I also owned a lot of international GDI features (the component that draws text and images to the screen). There were a lot of interesting issues drawing and printing Asian ideographs – I worked a lot with internal (e.g. Office) and external customers to isolate bugs they had run into on windows 95 so we could determine how (or if) to fix them for windows 98. I learned a ton about fonts – at one time I could tell you everything about how a TTF file was put together and explain details of font smoothing for hours. Today I remember only the basics, but I remember lots of reading and experimenting. Eventually I ended up owning GDI and User (user is the component that does windowing, menus, and controls) for all languages of win98. It made me learn a lot about how windows applications work, and a lot more about application compatibility issues (many app compat issues were rooted in the GDI and User components). Eventually (spring, 1998), windows 98 shipped. It was supposed to be the last version of “consumer windows” ever, so the team disbanded, and I got a job working for a pretty cool guy named Brad.

One of the things I love about Microsoft is that you can change jobs completely without leaving the company. The bulk of Brad’s team tested USB on NT5 (later named Windows 2000). A few others owned testing some of the core USB devices and related peripherals. I was in the latter group and tested video capture devices (aka cameras and TV tuners). Most of my tests were written via DirectShow APIs, so it was an opportunity for me to work on COM and C++. After a while, I really got the hang of it. One of my personal biggest successes at the time occurred when I discovered that I needed a generic library function – something that DShow probably should have provided, but didn’t. I wrote the helper function and it worked well for my testing. Of course, just a few days later, DShow added the API so I didn’t need mine anymore. The really cool part was that my implementation was almost completely identical to the API version – it made me feel like I knew what I was doing. Of course, there should probably have been better communication so I knew that the change was coming, but oh well.

I also did stupid stuff. One of the things you have to do if you’re playing with video and audio via DShow is connect all the darn pieces together. It’s sort of like this:

HRESULT hr;
hr = CreateAConnection(...);
if SUCCEEDED(hr)
{
    hr = CreateAnotherConnector(...);
    if SUCCEEDED(hr)
    {
          // connect the things, then if they connect, connect 
          // more things, etc.

. . .

I would end up creating what I called “the wedge” – where the inner if in my big connection thingy would start on column 70 or something like that. It made sense to me at the time, but … seriously, yuck.

Firewire/1394 video cameras were new at the time – I had one in my office that I could control via an app I wrote (it eventually became the 1394 camera SDK app). I also had this really cool device that could emulate television signals. I had a lot of fun with that one. This was also the first time I played with code coverage. Even then, I realized that the biggest benefit of CC was finding holes in my tests.

Brad also assigned me two “special” projects. One was some work for Windows 98 Second Edition (win98SE). I wrote (ported, actually), the power management utilities from Windows NT to 98SE. It wasn’t really that hard, but it was cool to write software that went in the box rather than just tests. The other project was this thing called Pandora. Pandora was a DVR (although I don’t think that term was even around yet). I think Tivo was announced at the time, but not out yet, so this was pretty cool stuff at the time. I was the only tester on the project and it was a blast. Unfortunately, however, for reasons I was never told, the project was cancelled.

Then, less than a year after the win98 team disbanded, marketing pushed hard for another windows 9x release. Phone calls were made, and before I knew it, I was on the Windows team again working on Windows Marketing Millennium Edition.

New Years eve, 1994 was my last day at my previous company, Midisoft. I worked as a bike messenger after graduate school (I was burned out and needed a break). Pedaling a bike all around Seattle every day seemed like a good way to burn off the drain of  graduate school, but after about 3 months of high-risk minimum wage exercise, I bought a paper and decided to read the want ads. I read every single want ad, circled half a dozen or so and  sent off some resumes. The only place that called me back was Midisoft. They wanted someone for technical support. I had learned a lot about making music apps work on Windows 3.1 while in graduate school, but that was about the extent of my computer knowledge. After a day of interviews, followed by a “mock tech support call” with the CEO and his wife, I accepted the 8-dollars-an-hour job and became employee number 17. I was on the job for a few minutes before I found out I was also going to be one of the company’s primary software testers. A few hours later, I also found out I was the new network administrator.

I learned a lot in the following 15 months. I replaced our OS2 server with Windows NT 3.1. I learned enough to get backups working properly, and to ensure that email and network printing worked (at least most of the time). We bought an accounting program that was only supported running from a NetWare server, so I learned enough NetWare to keep things running smoothly (my NetWare knowledge would eventually be both the reason I left Midisoft, and the reason I was hired at Microsoft). I remember getting a call on a Saturday because our NT server had a hard drive crash (it had one hard drive, and it died). I replaced the hard drive, then reinstalled NT (from floppies). Then I restored from the tape backup – but discovered that I only had incremental backups for the last few months. The good news was that I had backups, but the bad news is that it took me 4 hours to restore each of the individual  incremental backups.

I also learned a bit about programming, and quickly learned that I loved programming. I had written hundreds of batch files and doskey macros over the few years I had been using a computer, but being able to create usable software was a rush for me. One of our best programmers, Chris Fox, gave me a book on C (which I still have), and told me the “three most important things about programming in C” (answer: “pointers, pointers, and pointers”). I also began using a beta of a program called MS Test (MS Test later became MS Visual Test) that came with our MSDN subscription. I began automating a lot of tests, and learned how to write UI tests that would run on our localized software as well. I got good enough at the language that I began to write our setup applications for our products (the languages for the SDK setup app and MS Test were both based on Basic). I also ended up writing our bug database front end in Visual Basic.Somewhere along the line, I got a raise (to 27k 1994 dollars), and also got to move into my own office.  I was having a blast.

But then it got sort of weird. For the second quarter in a row, our CFO asked me to “adjust” the date on our NetWare server so that we could fit a few more days of sales into the previous quarter (did I mention that this was a public company). I could see other signs that things weren’t going well, so I went back to the want ads and send out a few more resumes. This time, the reaction was different – I didn’t realize what a year of software testing experience was good for, as I received five phone calls before I left for work the next day (I also learned about “contract” work and dealing with agencies). Over the next few weeks, I had a few interviews, none of which went particularly well. One day I went to “a dentist appointment” and spent the next 4 hours interviewing for a job on the windows 95 team. They wanted someone with testing experience (check), experience with NetWare (check), and experience testing on Asian languages (check again). They gave me an offer the next day for a contract position testing networking components on the Asian versions of Windows 95, with a start date of  January 3, 1995. I gave my notice the next day and let them know my last day would be December 31st.

My new job was much more lucrative ($20 an hour  -  but I did have to pay for my own insurance). In hindsight, my first team didn’t know that much about testing, but at the time, it was exciting. If you’ve  read hwtsam, this is the team where my manager told me “we don’t have time to automate, you just need to run these tests everyday”. Despite the “scripted-ness” sound of this, we were encouraged to explore as we tested, and I did. (note that I also went ahead and automated all of the tests my manager gave me in my first week). I really fell in love with testing at this time, and it was an exciting time to work at Microsoft.

I was a contract employee at this time, and knew nothing at all about how Microsoft dealt with contractors. All I knew was that I was on a 5 month contract, and that the contract expired on June 3. I didn’t realize that MS almost always renewed these contracts, and my manager (who was probably the worst part of the job) didn’t tell me anything. So, I did what I thought was the right thing to do and began looking for another job. Luckily, I found one –I interviewed at a place called Software Testing (ST) Labs in pioneer square in Seattle. One huge draw of the job was that it was a 10 minute bus ride from the condo where I was living. I talked to a few people at ST Labs (including James Bach), and got an offer (for another contract job) right away. The next day, I “reminded” my manager that my contract was up and that I found a new job. He looked a little distraught, but didn’t say much as I headed back to my office to work. Within 10 minutes, his manager was in my office telling me how much he liked my work and asked if I would entertain a full-time offer from Microsoft. I put on my best poker face and said I would listen, but I had already made a verbal commitment to another company. The company must have been more nimble in those days, as a courier showed up at my condo within a few days with an official offer from Microsoft ($40k, a $4k “signing bonus”, and some ridiculously low amount of stock).  I accepted the offer and made the awkward call to ST labs. They made a counter offer, but MS was where I wanted to be (I just didn’t know they wanted me).  On June 6, 1995, I became an official “blue-badge” full time employee of MS. I knew I made the right choice when ST labs moved out of their downtown location just a few months later.

Because of the way I was hired, nothing changed other than my badge color. I was supposed to go to New Employee Orientation, but they put the wrong date on my offer letter. The suggested that I come back the next week, but I just skipped it. I stayed in the same office and did the same work – nothing really changed.

At least not for another 4 months or so.

More later…

I expect that 2010 will be a year of change for me. I’m going to try some new things and put other things behind me. I’m still on task for a keynote at star east, but I expect that I will be speaking at conferences less than I have over the last five years. I haven’t an opportunity to “hang out” in the worldwide testing community as much as I would like to recently, but I’m going to find a way to make this happen and keep my day job. This is also probably the year where I get my personal life back on track – the last band I was in broke up over 4 years ago, and I’ve hardly played since then. Playing music is one of the few things that keeps me sane these days (running and soccer are the obvious others). I need to be more sane this year, so getting my horns dusted off and finding a new band are high on the priority list.

Did I mention how much I enjoyed the time off recently. I’ll be taking some more time off this year – not sure when yet, but I see an extended vacation in my near future. The last two weeks have been good for my body and soul, but I know I need more. I’ll update you as I figure out how to do it all.

Stay tuned.

Since then, the Chinese version was released, and the Korean version is imminent, and I’ve traded writing on weekends and evenings for more time with my family (and occasionally, more time for work). When I finished writing the book, writing another was the farthest thing from my mind, but since then, I wrote a chapter for Beautiful Testing, and have at least entertained the idea of writing something else…someday.

In hindsight, there are many things I’d like to redo with the book, but it is what it is, and I can live with that.It’s a book full of information and stories about how testers at Microsoft do their job. It’s a book about people, approaches, and some tools. It talks about when and why we automate tests, but covers a wide range of other topics as well, and I’m happy with the story it tells.

I think the book has sold somewhere around 5-6k copies (I haven’t looked at numbers in 6 months, but I’ll update this post if I do). That’s certainly not a huge number as far as books go, but it’s still amazing to me. My thanks go out to everyone who bought a copy (and more thanks to those who actually read it).

Let’s say, for example, that you want to make a change in your organization (I’ll leave the exercise on how to change the world) for another post. Just for fun, let’s say you would like to do a lot more Exploratory Testing in your organization (bad joke removed – should have picked a different example).

What do you do first?

The number one answer I expect (and I’m usually right) is that you need to convince management that ET is great (or that you should do more of it), and they’ll make a top down decree, and everything will be unicorns and rainbows(tm).

Bzzzt!

Management is one faction, but there are more. What if the rest of the testers on the team don’t see the value in ET? What if they don’t know how to do it? What if your customer demands that you only deliver automation results (or something else silly). What other factions can you identify? You need to think about everyone with an interest in the results – then take time to understand where they’re coming from and how the change impacts they’re thinking. With any change, you have gains and losses. What do you gain by doing more ET (note: also define “more”). What do you potentially lose by doing more ET? Top down edicts rarely work, so if you want a chance of success, don’t start there. Identify your factions, and come up with a strategy for working with each of them.

The important thing to remember is that you don’t change the process, you change the people. If you don’t think about how change impacts people, you will probably fail. Whenever I’m dealing with a performance gap, the six boxes model helps me think about how change happens.

The six boxes model (sort of based on Maslow’s hierarchy of needs) is a model to help think about all of the factors that go into change.

Box 1 is where management can help. Defining the expectations and feedback loop for the change helps people understand what they need to do.

Box 2 pertains to tools (e.g. sysinternals.com tools), and resources (including computers, a quiet place to work, etc.).

Box 3 is where most change efforts fall short. This is the “what’s in it for me” category. Prizes, bonuses and other material rewards fall into this category, but it can also (and often more effectively) be some other type of reward. The points awarded on xbox live or on stackoverflow are a form of box 3 reward. Done really well, box 3 can be satisfied by making the job more fun and interesting, and creating higher quality software.

Box 4 deals with the skill gap. For our example, it is the plan for how to teach or demonstrate necessary skills for ET, and may include instruction, reading material, coaching, etc.

Box 5 is about the people on the job. Are they capable of carrying out the necessary tasks? If not, you probably won’t be successful.

Box 6 is dependent on the other boxes. Binder says that if the other boxes are positive then this one is positive. My view on box 6 is that box 6 is free will – and you don’t mess with free will. Sure – keep box 6 in mind, but don’t f with it.

Now that you’ve thought of your factions, and mapped out the human element of the change, you’re just about ready to go. Before you start, remind yourself that while you have a plan, and you’ve anticipated as much as you can, things will change. You need to adjust your plan (revisit the motivations of your factions, examine your six boxes evaluation as you learn more information – hey – this sounds like “exploratory leadership)). Organizational change is often a moving target and being ready for that will help you be successful.

If all of this sounds like too much work, there’s always plan b – quit and go shopping.

Once upon a time (you know – college), I had long hair – long enough that I could reach behind my back and touch it. I kept it long until about 1999 or so. One day I got up, decided I was done with long hair and got it cut off (not all of it – yet at least).

As of 2005 or so, I looked sort of like this:

A few years later, my job in EE, along with prolonged exposure to small children (oldest was born in 2004) turned me into this (2007 or so – this is, incidentally, the pic I used for hwtsam, as well as my STAR East bio)

I had a problem last spring. I needed to get my hair cut, but couldn’t find the time. It bugged me so much that I took matters into my own hands and cut it all off. For the entire summer, I looked kind of like this.

A few weeks ago, I determined that hair actually provides warmth during cold weather. Today, I look like this:

I expect to have something resembling a full head of hair by the time STAR rolls around.

And now, you’re caught up. My head fashion show is over.

Before you eagerly read on, please remember that I’m not an expert writer. I have some experience, but I don’t claim to have it all figured out. I think it takes much more than a handful of magazine articles and part of a few books under your belt to claim you actually know how to write or are anything remotely close to an expert, so please take my advice with a grain of salt. That said, there were a few other techniques I used when writing both hwtsam and my chapter in Beautiful Testing that help me a ton when writing.

First – if you want to write, dedicate some time for it. I found that I needed at least an hour to be effective, and would often block off 2-3 hour chunks of time on weekends. In one prolific stretch of writing, I took a week off of work, and wrote every day from 9-12 and again from 1-4. In my hour “off” I would go for a run and grab some food – the break was very energizing. I think I wrote 3 full chapters of hwtsam that week.

My next trick was to employ the (10+2)*5 trick (I wrote a windows sidebar javascript applet to help with this). The idea is simple. Write for 10 minutes – don’t stop. If you feel blocked, move to the next section (see my last post on iteration for more details). Refuse to be distracted by anything during those 10 minutes. When 10 minutes is up (my applet sounded a bell and changed color), do something else. Sometimes I would stare at the ceiling, sometimes I would check email, and sometimes I would glance at my rss feed. Even if you’re “on a roll”, stop and take a break. When the 2 minutes is up, repeat. Then repeat a few more times. Once you’ve gone through five of these 10+2 minute cycles, an hour is gone (and if your experience is like mine, you made a heck of a lot of progress). At this point you can launch straight into another hour, or you can do what I did and take an extended stretch and refill the coffee break. I have no idea if this will work for anyone else writing about testing (or any other subject), but I can tell you that I don’t recall ever having writers block – in fact, I sort of believe it doesn’t actually exist, but I’ll save that discussion.

Final tip (and also a sure writers block stopper) is something I picked up from Hemingway (not directly, but through the Paris Review). The tip is to stop writing every day before you’ve run out of ideas (I couldn’t find the actual quote where I read this, so bear with me). Think of this scenario: you’re cranking out some prose at the end of the day. You have a fantastic idea for a sample, story, or exposition – something that you’re ready to crank out and know what you want to do. My advice is to NOT write it. Instead, jot down a few notes that will help you get things going the next day and then call it a night. Otherwise, if you write your brilliant work at the end of the day, you’ll have to start from a clean plate the next day, and starting writing from a blank page is hard. I find it so much easer to get in the flow of writing if I already know exactly what I’m starting with in the morning. Sometimes a night of sleep also helps to vet the idea a bit more. Again, ymmv.

Hope there’s some useful stuff in here.

But I iterate everywhere. At work, I put together skeleton project plans. Then I slowly fix them and add deliverables and dates slowly until I have something that works. When I write music, I start with a basic structure – sometimes a melody, sometimes a rhythm, and sometimes a chord progression. I slowly plug stuff in, add and remove parts, and repeat until I have something I like.

I’ve found iteration most beneficial in writing. When I write seriously (as in hwtsam or my chapter in beautiful testing, or many of the articles I’ve written rather than this blog), I always iterate. I usually start by creating an outline, and making the outline headings the subheadings in the chapter. I don’t worry about coming up with clever names, I just make sure the order looks right. Then (either immediately, or in another “writing session”), I’ll start filling in some text below the subheadings. When I get blocked on one section, I stop and move on to the next section. Sometimes I only write something like “talk about cyz configuration testing here” – either because I don’t have the data I need yet, or more often, because I don’t feel like writing about xyz configuration testing yet. In a later session, I may make another pass, or I may focus on adding specifically to another section or two. I add sections and remove sections as needed. Eventually, I find (or at least try to find) themes I can link together. Finally, once I “think” it’s done, I close the file and come back to it in a day. Then I read it, ask myself “what the f…heck was I thinking”, and make edits and rewrites, save then close. Then I do it again. Then I do it at least one more time. Eventually, it “ships” and I’m done (but I seem to always find stuff I want to change).

I can’t imagine not iterating on any task with a semblance of complexity – but not everyone seems to be on board with my approach. I was talking with a colleague some time ago who was putting on a series of collaborative events at MS. I was eager to help, so I asked them for details – e.g. how long will it be, how will you break it down, what are the outputs and a few other similar questions. He answered, “the strategy doc is almost done, and when it is, we can start thinking about the execution”. Yuk – that just seems wrong to me. Perhaps I’m a cowboy, but this is another case where I’d rather settle on the basics, try it out, and adjust. Sure, you need a vision / strategy, but I don’t think you don’t need a 10 page doc written before you get some people in a room to work together.

Or perhaps I just need to plan more – or set up a pre-planning meeting to discuss the preliminary plan – but not likely.

Some examples:

• If you’re spouse asks “do I look fat|stupid in this”, the wrong answer is almost always yes. This doesn’t mean that answering questions directed by your spouse is wrong, it just means you need to think about the right way to answer this question
• While driving to work|school|the mall, it is completely possible for you to swerve into oncoming traffic or drive down the sidewalk. However, just because you can do this, doesn’t mean you should, nor does it mean that driving is dangerous and that you shouldn’t do it anymore.
• Right now, I have the ability to blow away a massive number of important documents (yes, they could be eventually restored from backup, but I could cause big problems). This doesn’t mean that I shouldn’t have write access anywhere on the corporate network, it means that I’ve been trusted to do the right thing and that I should honor that trust.

I get annoyed when I see testers dismiss things flippantly because it’s possible to do it wrong (and more annoyed when they choose other stuff to do wrong instead). It’s asinine to call something stupid because you can mess it up by not thinking, yet it seems to be common practice.

So here’s my advice for everyone. Do whatever you want until it doesn’t work for you. Find out what works and doesn’t work by hypothesizing, experimenting and thinking. Reflect on what you observe and how you interpret that observation. Use that knowledge to fuel more hypothesizing and experimenting. If you have to discard an approach based on this process, you did the right thing. If you failed to ask yourself why an approach is or isn’t appropriate to your context, it is you who have failed.

The above paragraph works for testers too.

Quick – which team won? Did the Fire edge Real Salt Lake, or dir RSL outlast the Fire?

If you read a bit more, you’ll see that “RSL will face the Galaxy in the 2009 MLS Cup”, so if you go with majority rules you’ll be correct, since RSL did indeed edge the Fire last night. Headline errors aren’t all that uncommon (e.g. Dewey Defeats Truman), so I don’t fault the news site at all. Unfortunately, a very close relative of error, the false positive, has been bugging the crap out of me lately, and this headline reminded me that it’s past time to share my thoughts.

Let’s say you have 10,000 automated tests (or checks for those of you who speak Boltonese). We had a million or so on a medium sized project I was involved with once, so 10k seems like a fair enough sample size for this example. For the purpose of this example, let’s say that 98% of the tests are currently passing, and 2% (or 200 tests) are failing. This, of course, doesn’t mean you have 200 product bugs. Chances are that many of these failures are caused by the same product bug (and hopefully you have a way of discovering this automatically, because investigating even 200 failures manually is about as exciting as picking lint off of astroturf). Buried in those 200 failures are false positives – tests that fail due to bugs in the test rather than bugs in the product. I’ll be nice and say that 5% of the failures are false positives (you’re welcome do do your own math on this one). Now we’re down to 10 failures that aren’t really failures. You may be thinking that’s not too big of a deal – it’s only 1% of the total tests, and looking at 10 tests a bit closer to see what’s going on is definitely worth the overall sacrifice in test code quality. Testers in this situation either just ignore these test results or quickly patch them without too much further thought.

This worries me to no end. If 5% of your failing tests aren’t really failing, I think it’s fair to say that 5% of your passing tests aren’t really passing.  I doubt that you (or the rest of the testers on your team) are capable of only making mistakes in the failing tests – you have crappy test code everywhere. A minute ago, you may have been ok with only 10 false positives out of 10k tests, but I also think that 490 of your “passing” tests are doing so even though they should be failing. Now feel free to add zeroes if you have more automated tests. I also challenge you to examine all 9800 tests to see which 490 are the “broken” tests.

Yet we (testers) continue to write fragile automation. I’ve heard quotes like, “It’s not product code, why should it be good”, or “We don’t have time to write good tests”, or “We don’t ship tests, we can’t make it as high quality as shipping code”. So, we deal with false positives, ignore the inverse problem, and bury our heads in the sand rather than write quality tests in the first place.  In my opinion, it’s beyond idiotic – we’re wasting time, we’re wasting money, and we’re breeding the wrong habits from every tester who thinks of writing automation.

But I remain curious. Are my observations consistent with what you see? Please convince me that I shouldn’t be as worried (and angry) as I am about this.

It rarely works. They inevitably follow up with, “what do you do there?” – which, for better or for worse is a much more difficult question to answer. Depending on their technical knowledge (and my mood), I’ll say something between “I work on a team that does technical training, internal consulting and cross-company projects for engineers”, “I’m the Director of Test Excellence, and “I stop people from being stupid”. It was much a much easier question to answer when I worked on a product team, but I like the job, so I’ll deal with the moments of awkwardness.

I thought I’d write down a longer answer for those who are curious (or want to help me with a better definition).

The biggest thing I’m working on this year is helping engineers across the company have a common concept of software quality. This includes working with marketing on customer perception of quality and a lot of talking with people from around the company to see which practices are common, and discover some practices that should be shared more widely. It’s a hard problem to solve (and there are a lot more pieces to it), but it’s a fun challenge.

I’m also working on a variety of small projects to increase collaboration among testers and other engineers at the company. With nearly 10,000 testers, there’s not nearly enough sharing of ideas, practices or innovation among people solving problems that are likely much more similar than people realize. Every time I see a duplication of effort or the same question asked on a distribution list for the 3rd time in a month I’m reminded of how much more work there is to do in this area.

Edit: I forgot a big chunk worth adding

A reasonably sized chunk of our organization’s work is technical training for engineers at Microsoft. My team teaches some classes, but I work with vendors to teach and design a fair number of test related courses world-wide. I also own scheduling and prioritization of technical courses for what we call MSUS (shorthand for all MS engineers in the US outside of Redmond). It sort of a thankless job, but needs to be done and I don’t mind doing it.

The bulk of the rest of the time goes to what I call – “being Alan”. I organize and schedule meetings for our company-wide test leadership team and test architect group, and chair our quality and test experts community. I also function as chief-of-staff for my boss’s staff meetings (he attends the meeting alternate weeks, and I take care of the agenda and flow of the meetings every week). I participate in a few virtual teams / micro-communities (e.g. cross-company test code quality initiatives or symposiums on testability). I’m on the board for sasqag, and put in a few hours a month keeping things alive on the msdn tester center. I give talks to internal teams a few times a month and mentor half-a-dozen testers in various roles around the company. Finally (and most importantly), I manage a team of 6 people who work on similar projects, as well as teach testing courses. It helps a lot that the team is so smart and so motivated, because I’m most likely not the best manager in the world.

Beyond all that, I spend probably too much time staying connected with what’s going on in the testing world outside of MS. I blog a few times a week, speak at a few conferences a year, and tweet once in a while. It can be a balance problem some times, but I think it’s important enough to make a significant effort to keep up.

There’s probably more, but I think that covers most of it. Now you know.

I’ve been writing less lately while I turn my attention toward my often neglected day job. I have a few projects on the horizon, and I’ll add them to the list if (or as) they come to fruition.

I received my copy over the weekend (much to the annoyance, I’m sure, of several other authors who are about to rebel against the empire if their copies don’t show up soon). I’m happy to have mine, and although I read the entire book in digital format, I’ve been flipping through it off and on for the last 3 days. I’m thrilled to be a part of it, but I have to tell you that I’m more excited after reading it again and finally holding it in my hands. The variety of information, styles, and knowledge is fascinating – each one opening up different possibilities and questions to ponder. It’s a fun read that I hope you check out. Best yet, the proceeds from the book all go to buying mosquito nets to help prevent malaria in Africa – what a great opportunity to get some practical testing advice and help out those in need!

As all gainfully employed workers in the tech field will tell you, we all have side jobs as tech support for all members of our immediate and extended families. This weekend, my mother-in-law opened a support ticket with me regarding her laptop – it was crashing randomly (that’s all the details you get when your m-i-l opens a support ticket).

So – I turned on her laptop, let it boot, then dealt with message after message from applications starting up and telling me stuff I didn’t care about. A backup program telling me that it needed a product key, an external hard drive utility telling me the drive wasn’t connected (duh), and an OEM replacement for windows wireless config launching to tell me I’m connected to a wireless network. The experience was annoying. But there’s a bigger problem. As I was looking at the 3 different web browsers installed and the few dozen or so other random programs and utilities installed, my first thought was “no wonder she’s having computer problems – she’s installed every app under the sun”. I always try to keep my main work machines somewhat “clean” – only installing applications I consider tried and true for worry that they’ll mess something up. Then I realized that’s wrong – I should be able to install whatever the hell I want without fear of losing overall quality (who knows – maybe I can and it’s all a mental problem on my end). The point is, that we (computer users) don’t seem to expect software to work. We’re not as surprised, alarmed, or pissed off as we should be when software doesn’t work correctly. Honestly – I’ve belittled people in the past for calling things bugs when they’re 99.99% user error, but I was wrong – user error or not, that .01% matters.

Ok, so software sucks. It really doesn’t matter – it’s still a profitable industry. That’s true, but I wonder how long it will be true. I wonder if something horrible (even worse than Windows ME** :}) has to happen before the world demands higher quality software. My hope is that we can start making better software long before something like that happens.

Oh – as far as my mother-in-laws computer goes, there was a crash dump on the machine. I attached a debugger and poked around a crash in the wireless driver. I put a later rev of the driver on the machine and so far, so good. I hope it stays that way…for at least a little while.

Want to improve v2? Talk to customers of v1 and ask which of the ilities they suffer the most from. And/or talk to people who didn’t buy your software and ask them which ility chased them away. Those are the ones that count.

Perfect answer, but keep in mind that the questions you ask are critical. You can’t ask “do you want the product to be more reliable”, or even “from this list, choose the one you care about the most”. For former question will always result in a “yes” answer, and the second will probably just result in confusion. Instead, you can ask questions like “tell me what you like most about the software (or what you dislike the most”. Ask open ended questions and take notes – take lot’s of notes. After  you’ve talked to a good sample of customers, break the notes into individual comments and start sticking them on a wall. Look for affinity – start grouping items and looking for themes. Then, see if an ility aligns with a theme. Eventually, you’ll have a bunch of big fat quality bulls-eyes on the wall waiting for you to address.

I have one minor nit where Adam missed the mark – you don’t have to wait until v1 is out to collect this data. If your software team is worth their salt, they’ve defined the customer segments they care about far before v1 hits the street. Interview customers from that segment and ask them questions like “this product does foo, what do you expect a high quality product that does foo to do?”, or “what would make you want to use a product like this?” or “what would make a product that does foo unusable for you?”

The fun part is that I’ve sort of done this (in a very general way), and have a list (that certainly won’t work for every piece of software in the world, but is worth discussing). I haven’t yet figured out how to push the dial on these ilities, but but that’s what I’m going to try and figure out – using this blog as a sounding board while I think.