Archive for the ‘Communicator’ category

Fuel for work

August 26th, 2010

Due to some unforeseen issues, my experiments in remote work are continuing. These days, the situation is a bit different though – I’m working in the same time zone,and I’m showing my face at work one to two days a week. Of course, my observations and musings about working remotely continue.

The days that I’ve been at the office have been packed with human interaction – group meetings, one on one meetings, interviews, and hallway conversations.Meyers-Briggs tells me that I’m an introvert (and they’re absolutely right), but after being away from my coworkers so much lately, it was something I really craved. I also noticed that I generated a lot of work items out of these meetings. However, people didn’t give me work to do or ask me to do anything, but through the interaction, I discovered work that I needed to do. I realize that this is unique to my role on the team and that some remote workers just want to be left alone, but for me, it was energizing.

I should note that my job is a lot about discovery. I am trusted to figure out what I should be doing – or prioritize from a list of things I should or could be doing – then do those things. I reprioritize every day (sometimes multiple times during a day, and have a pretty good success rate at picking the right things to work on. It’s a great role, but it’s definitely not for everyone.

So, here I am away from the office again, but my todo list is absolutely packed. And that’s as good of a reason as any to write a short blog post today.

But not until I share this comic from The Oatmeal on the subject (because it’s funny and relevant).

No comments »

Posted in Communicator, Me, Test Careers

ET and Me

August 15th, 2010

I’m a big fan of exploratory testing. I’ve used the approach long before I knew what it was called and think it’s the core of good testing. it’s so ingrained in the general approaches I’ve used in my career that I often don’t differentiate between exploratory testing and plain-old-testing (I’ve gone as far to say explicitly that all good testing is exploratory in nature, but as with most times I’ve used the word “all”, I’ve found exceptions).

I’ve been working on my ET skills for over 18 years, so it’s nice to see so much recent emphasis on the approach in blogs and books. As I said, I think it’s the core of good testing, so a strong foundation in ET can only help testing improve overall. As with any hot topic in any field, there are many strong advocates, and there are definitely “camps” of thought on what exactly ET is. On a side note, those of you who know me, know that I am certainly not afraid to take jabs on just about any of the subjects I’m passionate about – both in person, and in this blog. I once made a small (a few words) comment in a blog post about ET and “belonging to a club” that brought down a deluge of email reactions that I still ponder frequently. To this day, I both regret the remark (it was a thoughtless jab), and remain somewhat dumbfounded by the reactions to the comment.

Anyway…I recently introduced ET to my (still sort of new) team at Microsoft. Actually, I didn’t really introduce it as much as I revealed it, as I discovered a natural talent throughout the team that went far beyond what I was able of teaching them.

But let me back up a bit…

I have given several presentations to my team on a variety of testing topics. I’m a big believer in testers having a big “toolbox” and knowledge on when and where to use those tools. I noticed early on in my time on the team that some testers would get so caught up in “running tests”, that they forgot to engage their brains and think about what they were doing. I also noticed that although most testers were experts in their feature area, that some had limited knowledge of the rest of the product. So, at a regularly scheduled brown bag (lunch time tech talk), I gave an intro / overview on ET. To follow up, I asked if there were any volunteers who would like to take part in an ET session with me to practice (with the secondary goal of learning more about the overall product). I quickly had a group of four volunteers, so I set up a time, and we were off to the races.

The format was:

90 minute meeting.

I took the first 10 to talk about our goals for the meeting (Learn ET, Learn about the product, and Learn about tools that may help us with ET). Finding bugs is a probable, but not necessary side effect of these goals.
For the next 75 minutes, we tested together.
Last 5 minutes was a quick debrief and sharing of thoughts.

I took notes during the meeting on what we discovered (and then wrote them up for distribution afterwards).

At the first session, I didn’t know at all what to expect. I didn’t know if people would learn, and didn’t now if we’d find bugs (I worried that if we didn’t find bugs that people wouldn’t see it as successful). I was also worried about engagement – what if people got stuck and “checked out”?

It turns out that I didn’t really need to worry. The room rocked with engagement. We agreed on random place within the application to start, I threw out a few ideas, thought out loud for a moment, and before I knew it, I couldn’t keep up with the comments and bugs and excitement in the room. I was blown away how well it went (again, it was nothing to do with me – I just pointed them in the right direction).

Based on the success, I tried another session. The same worries came to mind. The first session went so well that I had a high bar to live up to, and figured that it may have been due to the “early adopters” who signed up so quickly for the first session. Once again, I was proven wrong as a completely different set of people filled the room with ET energy.

Since then, I’ve moderated four more sessions (including one via teleconference ) and have had similar results in each of them. Better yet, some of those attendees have conducted their own ET sessions within their own team (and had similar success). Individuals are also using the approach outside of specific sessions.

Some points to share include:

We’ve had 3-4 attendees (plus me) for each session, and I’m pretty happy with the learning experience. I think this size group is big enough that people can learn from each other rapidly, and small enough that everyone gets to be heard. I also don’t think I could keep up on notes with a larger group.
The session length also seems to work well. People are engaged throughout and there’s barely a slow down before the session ends (one attendee mentioned that they felt guilty because the session didn’t feel like work!).
Everyone should get used to thinking out lout – especially early in the session. It helps people learn and build off of each other’s ideas
The focus on learning has worked well for us. I think that anytime testing focuses on finding bugs, that it veers off track

And that’s pretty much it. We’ll definitely continue to hone our ET skills, and I’m sure we’ll continue to experiment, add to our skill set, and tweak our ideas and processes as we go.

2 comments »

Posted in Communicator, Software Testing

An Approach to Code Coverage

August 9th, 2010

My team at Microsoft is starting to use code coverage a little more diligently. Code coverage has been used for some time on the team, but we’re just now getting to a common approach and recording mechanism. There are a few things about our approach that are a bit different than I’ve seen most other people use, so I thought I’d share it here.

As a quick aside, we measure code coverage using block coverage (block coverage is similar to line coverage except that it groups continuous non-branching statements into a block for counting purposes). I’ve found that bullseye has a very good primer on code coverage measurement for those who are interested in more than I feel like writing about today.

A small group of us worked on the toolset, process and strategy for our team’s use of code coverage. One topic that came up was what target percentage of code coverage we should set as a goal. Years ago, when I was on the CE team, I helped build and deploy our code coverage toolset. Once our tools were working and we measured coverage, we set a goal for the next release (65% if I remember correctly). We cranked up the number a few percentage points each release, and I think we were approaching 80% by the time I left the team. As I type this, I remember something about making a promise to shave my head if we got higher than 80%. That’s only funny because about a year ago, I did shave my head (it’s since grown back).

That history is interesting, because I had been anticipating the question (target for code coverage percentage), and I was (and am) adamant about the code coverage goal I would like our team to shoot for.

I’m adamant that we have no goal for code coverage.

The problem of having a target goal for code coverage is that code coverage will improve – i.e. you get what you measure. Wait a minute – shouldn’t improving code coverage be a good thing? It is (for reasons I’ll explain lower), but here’s what usually happens in practice.

Let’s say I own three features – one major feature with high customer impact, one area with low customer impact, and one that’s somewhere in the middle. Now, let’s assume that my team has a goal of 70% code coverage (measured as an average across the components I own). I measure code coverage, and here are my results.

	Feature 1 – High Impact	Feature 2 – Medium Impact	Feature 3 – Low Impact
Code Coverage	70%	60%	50%

Now, if your goal is to get your average to 70% (or even if your goal is a minimum of 70%), you are going to put your testing efforts into testing medium and low impact areas. Shooting for a code coverage goal can convince testers to throw out their prior knowledge of risk and impact, and instead focus on “improving the number” – by testing stuff that probably doesn’t need more testing, while ignoring potentially important stuff. If you never measured code coverage in the first place, would you really spend time doing additional testing on the medium and low impact areas of the product (with more effort needed for the low impact area)?

It’s time to discuss the goal of measuring code coverage. It’s foolish to think that higher code coverage has anything at all to do with product quality. It only measures whether code is executed on at least one path. One of my common “Alan-isms” is this:

The only thing that 80% code coverage tells you is that 20% of your code is completely untested

What code coverage does do is point you to holes in your testing. The goal of measuring code coverage is helping you (as a tester) understand what is not being tested. Once you discover what’s not being tested, you can make a choice based on risk and impact on whether you need to add additional tests – or if your time is better spent elsewhere. By not having a percentage goal for code coverage, I hope the team can focus on improving tests and testing rather than improving a number.

By the way – you can replace the words “code coverage” with “test automation” above and tell a pretty similar story.

Another thing we’re starting to do is measuring code coverage on check-ins. We’re fairly late in the cycle now, so we want to be careful of regressions and ensure that all of the code coming into the product is well tested. What we do is filter our code coverage view to only look at changed lines of code in the changed list. Say we have a binary that has 10k blocks of code, but the latest checkin only changed (or added) 25 blocks. We can filter our testing and coverage on just those 25 blocks and ensure that we’ve at a minimum executed each line at least once during our initial testing. This helps a lot with regressions, as we can ensure that we look at error cases and other little used paths very close to the checkin rather than waiting until those errors pop up at a much later date.

This gives us the potential for 100% code coverage on every changed line of code.

But I would never make that a goal :}

5 comments »

Posted in Communicator, Software Quality, Software Testing

Getting it right the second time

March 31st, 2010

I’ve been working on integrating a set of code scanning tools into our build system. Test code needs to be trustworthy – when test code fails, it should indicate a product failure. More importantly, when tests pass, they better indicate that the functionality under test works. One part of increasing quality is getting rid of the coding errors that are most prone to causing test failures. This includes a basic set of fxcop rules for managed code, similar rules for native code, and specific checks for hard coded paths, hardcoded passwords, and other easy to spot things that will eventually cause false positives.

The framework we use requires “hints” to find the reference libraries at compile time (basically, we run fxcop against every managed assembly at build time and track a full list of errors). The hints are just a pointer to the library in the project xml file. It’s a bit of a pain, but completely worth it in the end. I spent a few hours last week building a portion of the tree, reviewing the errors for the references the framework couldn’t find, adding them, then verifying the additions. Lather, rinse, repeat.

I made a ton of progress in those few hours, but stopped because it just didn’t seem efficient. I was doing other work during the builds, so I felt productive, but it just didn’t seem right. So, I stopped and did other work. As I look back on it, it wasn’t really even a conscious decision – I just sort of mentally reprioritized that task.

Today, for some reason, I decided to take another look. I thought about it for about two minutes and had a solution. It was so dirt simple that I’m embarrassed it wasn’t my first choice. Rather than build part of the source code, review the errors, find the binary, and add the hint, I took care of it all at once. When the framework couldn’t find a reference, it put an error in the log file something like this:

Could not find reference – Microsoft.Internal.Foo

It printed one of these for every reference it couldn’t find, so I just built everything – and while it was building, I wrote a script that would parse out the reference name, look for the binary on disk, normalize the paths, remove the duplicates (multiple projects will share references), then add the appropriate xml to the project file. Total time on task was about 20 minutes and it worked perfectly.

The point of all this is that I realize that I do stuff like this all the time. I often dig into a problem, pause (or leave), and then come up with an optimal solution. Part of me thinks that I should recognize the optimal solutions the first time, but another part of me wonders if I’d be able to come up with good solutions if I didn’t get my hands dirty first. I’ll pay more attention and let you know how it goes.

No comments »

Posted in Communicator, Software Testing

A Tour of My Office

March 26th, 2010

I’ve been in my new office for nearly a month, so I thought I’d share a tour.

For the most part, it’s a typical Microsoft office – about 100 square feet with a window, desk, door and …stuff. Here’s what you see when you walk in the door.

I have three computers. A HP Z400 drives the two monitors in the foreground, and a Dell 670 drives the remaining LCD. Both are dual proc Xeons (which appear to windows as 4 processors), the the HP is younger by about 5 years (it’s brand new) and builds about twice as fast. I use the Dell for mail, charging my phone, writing, and most everything else that isn’t build, analysis, or test related (although it can cover those tasks if the Z400 is in the middle of something. I also have my trusty Lenovo x200s as another backup. Most of the time it also functions as my primary “home” work machine.

I have a lot of books – here’s one of my bookshelves.

I could do a whole other post explaining where some of these came from. On the top shelf (next to several copies of hwtsam), are my “commemorative” copies of Windows 95, Windows 95 Plus Pack, and some other software I worked on. The other day I ran across some “gold” copies I have of Win95, and the Win95 SDK/DDK that were handed out to team members when we shipped (it’s just a regular CD printed on a gold colored disk in a jewel case).

Here’s a reverse view of my desk – pretty much the view I have now while I type on the Dell.

Note that the phone on the right doesn’t have any keys – it connects via USB cable and works with communicator. It’s the phone that rings when you call my office (assuming I’m logged into communicator on that machine).

Here’s a view of my other bookshelf.

Again, lot’s of stuff that I’ll explain if you ask.

Finally, here’s some crap piled near my window – at Microsoft, you tend to accumulate stuff that is important enough to keep, yet there’s no place to keep it at home.

In this picture are two “Ship-It” plaques (one is filled), my 10 year award plaque, a Windows 95 team commemorative clock, and a plaque containing the one patent I have filed (the block next to it is a patent “cube” for the same thing – you get that for successfully filing for the patent.

The one thing I probably miss the most from my old office is the lack of writable surface. Just about every vertical space (and one of our tables) in the old TestEx team room was a writable surface, and I got used to writing diagrams, notes, etc. everywhere. For now, I just have a little whiteboard, but I’ll try to see if I can get it replaced with something much larger at some point..

So – there you go. If you want something explained or have questions, let’s chat in the comment section (or on twitter if you prefer – @alanpage)

8 comments »

Posted in Communicator, Me

Checking in on the new gig

March 22nd, 2010

Today was my 14th (business) day on the communicator team. I’m into sort of a groove and having a good time. My job today is so drastically different than what I was doing just a few months ago that I thought I’d share a few things so everyone would know what I’m up to.

I’ve managed to initiate a few cool projects and get myself injected into a few more. One thing I really love about this team is the amount of trust my manager and teammates have in me and my ideas. Last week, for example, I was pointing out some perceived flaws in one of our processes – along with my ideas of how the processes should work.Thinking out loud, I said, “it would be cool to do a case study and gather some data on this.” I already have the details and participants lined up and should have the study done in the next week or so. It’s really exciting to see things more so fast.

Another principal sdet on the team and I are starting a program that will enable testers on the team to step away from the bulk of their day jobs for up to 4 weeks. This will enable testers to work on projects they may not get a chance to otherwise – and, of course, some work gets done that wouldn’t get done otherwise.

I’m also giving a series of brown bag talks on various testing topics. You all know that I’m a fan of the five orders of ignorance and this is a great opportunity for me to help people learn what they don’t know. I’m also heavily involved in the interviews for new team members – it’s fun to interview people for testing positions (in EE, I interviewed people to be teachers and advocates for test).

And finally, as a big proponent of test code quality, I’m unleashing a boatload of static analysis tools on the test tree. This is a nice project because it gives me a chance to do a cursory review of all of the test code in the tree – the end result will be higher quality tests and tools – tests and tools that can be trusted to provide accurate results. The topic of static analysis is worth another post someday – but I’ll wait until I’m done with this little project first.

I think that covers it – so far so good!

No comments »

Posted in Communicator, Leadership, Software Testing

What comes with the gravy?

March 4th, 2010

I mentioned a week ago that I was giving a talk on career tips called “Ride the Gravy Train”. I gave the talk today (which had a remarkably large turnout), and think it’s a talk I’ll deliver again (with some tweaks).

Without going into details, here are the tips:

do the right thing
try different
speak up
learn your A-C-B’s
know that you don’t
know who you don’t
follow the leader – lead the follower
don‘t flip the bozo bit
…and don’t burn bridges either
find a mentor
ride the gravy train
find the steepest learning curve
the three p’s
be happy
a career is a journey, not a sprint
there’s nothing wrong with self-promotion

As you can tell, there’s nothing Microsoft specific here (and probably nothing controversial either). As usual, of course, I use few words on my slides (this is the sum of all of the text), and the words I use are too obscure for most people to provide context.

Given that it’s my product now, I’m considering doing a live meeting version of this talk (and if that goes well, others). I thought about doing something like this before, but as soon as I suggested it, I got very, very busy. I’m still busy, but at least I can say I’m testing (and I promise to anyone who wants to listen to the talk that I will take every bit of feedback you give me on your experience with the product and get it to the proper people). Not sure when yet, but probably in a few weeks. I’ll post something here and frequently on twitter when I get things squared away.

1 comment »

Posted in Communicator, General

My first day(ish) as the new guy

March 3rd, 2010

Originally, I was supposed to join the Communicator team today, but a cancelled day of vacation pushed my start date up to yesterday…but there was a meeting I wanted to attend on Monday, so I’ve basically been out of building 21 (EE land) and in or near my new office since Monday afternoon. So, let me tell you’ what I’ve been up to since then!

It’s been a bit of a slow start – given that I started early and the pace that the hr backend is updated, I don’t have permission to enlist in the source code yet. On top of that, I’m told it will take a week or two to get my server account moved so I can use the in-development release. I’m also still waiting on my new computer (I still have my laptop and a computer that will become my secondary machine, so I’m certainly not completely blocked).

And, despite waiting for permissions, I don’t feel blocked in my ramp up either. I have a 3-pronged approach to ramping up on a test team that is working well for me so far. First off, although I normally don’t like going to meetings, I’ve already attended half a dozen or so. Some, I’m invited to, and others I just tag along. It’s a good way to figure out what’s going on in the project, learn faces, and get a sense of the team dynamic. Secondly, I’ve spent several hours reading through bug reports. It’s great – without talking to hardly anyone, I already have a sense of the types of testing approaches used on the team, a sense of how the database fields are used and interpreted, and even some insight into individuals on the team. I’ve recommended reading bug reports as a way for testers to ramp up quickly on a team for years, and I’m delighted to see that it works. And – I’ve read team wikis, test plans, and whatever other collateral I can find in order to help round out what’s going on.

I’ve been reading so much, in fact, that by Tuesday afternoon, my eyes were aching and my vision was fuzzy. So fuzzy, in fact, that I visited the optometrist on the way home for my first ever eye exam. Shortly later, I ordered my first ever pair of eyeglasses. I’ll post photos of my new look as soon as they show up.

Today, I made sure I took a few breaks to rest my eyes – I used the time mostly to unpack a few items and tidy up a bit. My new office is now entirely unpacked except for a big box full of random cables that I’m afraid to get rid of because the moment I do, I’ll need one of them.

I think that’s about it. Oh – except that I’m giving a talk tomorrow on random career tips. I think it will be interesting, but we’ll see. Looking forward to sharing them here at a future date…or perhaps over live meeting???

2 comments »

Posted in Communicator, Me, Software Testing

Why Office Communicator?

February 28th, 2010

Since announcing my pending job change, several people have asked me, “why did you choose the Office Communicator group for your next position?” The answer is also part of the internal talk I’m giving this week on career tips, so I thought I’d share it here.

One of my favorite things about Microsoft is the wide variety of teams, products and roles available across the company. While considering this change, I looked seriously into possible roles on at least half a dozen teams. One job was extremely technical – working on testing a major new feature for a critical and complex component (sorry, I have to be vague since it’s unannounced). Another position was working on a cool technology helping them with their test tool infrastructure. Talking to the managers in charge of these teams helped me figure out what kind of position I was looking for (when I started looking, I wasn’t entirely sure what I was looking for). In fact, when Ross (my new manager) and I first talked, I told him I’d keep him in mind as a backup – but we eventually talked and I began to change my mind.

When looking for a new job, the 3 Ps are important to keep in mind. You can follow a Person (a manager you really want to work with), you can work with People (teammates can make or break your experience), or you can go to work on a Product or technology you’re passionate about (or ideally, a combination of the three). As I mentioned, I’ve known Ross for 7 or 8 years (Ross used to chair the Test Architect Group before I took over). I’ve also watched Ross doing some extremely cool things as a manager over the past few years. As I dug in a bit deeper to the org (you should always do your homework when changing jobs) I found more things I liked. Ross’s manager spent some time with me, and showed me that he understood more about the challenge of software testing than just about any other manager of his level in the company. I also discovered that a peer of his, as well as a team mate-to-be are people I worked with and respected in previous jobs. I’ve met various other people from this org over the years and I’m certain it’s a team I can work with. What surprised me in my search was how interested I became in the technology (it’s quite common to love a job where you aren’t passionate about the technology). As I talked to the team and learned more about the product, I began to get pretty excited about the potential – and felt pretty good that I found a position that nailed the 3 Ps as well.

Office Communicator is about a lot more than an IM client (in the past, I have jokingly referred to the product as “messenger – enterprise edition”). There’s potential to do some pretty cool stuff with online meetings and recordings (and thing in the client too). There’s also plenty of more technical challenges below the surface (distributed algorithm testing anyone?), so I’ll get to do a wide variety of work. I think it will be fun working on a team like this and hope to have a lot of fun.

And, beginning Tuesday, I hope to begin to figure out much more about this team and start to figure out exactly what that wide variety of work entails. Stay tuned.

2 comments »

Posted in Communicator, Me