Author: Alan Page

Today I saw yet another info-byte on the perils of multi-tasking. I too, think that multitasking is rarely (if ever) as productive as multitaskers believe it is (note: my favorite study compares bong hits with multitasking…ymmv) – but we’ve all seen stuff like this over time.

What’s missing from many of the studies and stories that I’ve seen is a definition of what they mean by multitasking. The study above uses examples (and good ones, like the problem with texting and driving), but then seem to lump all other pseudo-multi-tasking into the same bucket. If, by multi-tasking, you mean texting while driving, writing code while playing a video game, or walking the dog while juggling, the studies are 100% true – but perhaps less so, if you consider multi-tasking to be ordering a pizza on the phone while taking your dog for a walk, or listening to the radio while cooking dinner.

But for most tasks requiring brainpower, we don’t actually do two things at once – humans are largely a single threaded system, and we tend to do one thing at a time quite well – it’s the context switch that kills us. This is especially true in knowledge work where we’re writing / coding / speaking / thinking; and where the context switch of doing something different (checking email, checking a blog post, answering the phone, sending a tweet, etc.) ruins our train of thought and takes us out of “the zone”.

But sometimes, context switches are necessary. My favorite example (as anyone who knows me would guess) is cooking. When I cook any meal of significance, context switches are required. One cannot cook a meal serially. If I make the vegetable, then the appetizer, then the main dish, then the desert, I get to serve everything cold. I need to start with my deadline (dinner time), think through the process needed to make each dish, then determine what needs to be done in what order to accomplish my goal. If I make a marinade that needs to sit for three hours, I better do that first. Anything that requires heat should finish as close to dinner time as possible. I love the orchestration of cooking a great meal, but this task would be impossible (or at least a failure) if I didn’t rely on the context switch.The trick is to plan for the context switch and optimize – do the stuff where you know you need to wait first. Mentally work backwards and set up intermediate check points. Sure, doing only one thing would probably be faster, but it’s not always practical..

I apply the same techniques to my daily work. I look at all of the things I need to do (which is nearly always more than I can do), and do anything with a built-in wait first. For example, I start most days by sending out any requests for information I may need, things I need reviewed, etc. Then I take on bigger tasks. For writing tasks, I use a pomodoro timer – for coding or testing tasks I just go to work (I’ve found that the pomodoro beep ends up being a distraction from these types of tasks– but that’s just me). But regardless of whether it’s a pomodoro timer, or a natural break in flow, I tend to use those breaks to let interruptions decide what’s next. It’s a conscious task switch, but just like in cooking, where I may take a moment to look around the kitchen and see what’s going on (oops – better turn the oven down), if you’re in a role where you need to own multiple pieces of a project, it’s critical to allow yourself to step back, look at the big picture, and either dive back in again, or refocus on another area.

Now, to be fair, when I have a bigger project to take on, I’ll minimize multi-tasking as much as possible. I can think of several occasions where I needed to focus on accomplishing a specific task – even at the expense of potentially more critical tasks. My approach in those situation involves a pomodoro timer, and three-hour blocks of time in the morning and afternoon, broken up by breaks for email and task juggling. But – for the most part, I’ve found much benefit in strategic task switching and planning ahead than in giving up and admitting to stupidity just because your plate is full.

Last week, I posted a small testing challenge (in short, an application wouldn’t launch).Shmuel Gershon has a wonderful write up of how he found the problem here.

For those too lazy to read, the problem was that the application attempted to read a reg key on launch, and failed to launch (with no diagnostics) if the reg key didn’t exist.

Having done a massive amount of application compatibility testing over the years, this is a fairly common scenario I’ve run across (simplified quite a bit in this case). Typically, it works like this: a team makes an update to an application or operating system, and some application begins to fail in a bizarre way (including failing to launch, as well as functionality or stability changes). After some investigation, we usually discover that the application relied on a registry key, a file, or functionality that no longer exists, or has changed. Half of the time, the application vendor is out of business, or doesn’t have the bandwidth to make an update, so we have to figure out what went wrong and hack do whatever is necessary to ensure the application continues to work.

Some of the time, the investigation requires a bit of reverse engineering, but at least 9 times out of 10, I use tools like sysinternals procmon to see what’s going on. I was happy (but not surprised) to see that Shmuel ended up using this tool to find the issue (and was even happier to see how well the tool led him to the exact issue).

For my job, I do almost as much investigation into why things don’t work as I put into discovering what works (and doesn’t work). To me, the detective work is part of the testing role. I was surprised yesterday, when a tester told me that his job was problem detection only, and that the rest was for someone else. That world of testing would be boring to me – ymmv.

Congrats Shmuel for the great investigation, and for having this valuable tool in his toolbox.

 

Note: for simplicity, I put the key under HKCU so everyone could access it, and used an obvious key name (MagicValue) – in the real world, it’s not always this way.

Some updates and clarification below.
And a NEW HINT below too

Recently, I was discussing the diagnostic, debugging, and troubleshooting aspects of software testing, and this morning, while playing with Visual Studio 2012, I created a quick little exercise where the solution relies on these skills.

This zip file contains an app (theapp.exe), and the the vc 2012 runtime dll (both for 32-bit Windows) (note, this should be all that is needed, but if you get a load error, let me know in the comments or on twitter).

But even with the runtime, the app won’t launch. I’m sure that many of you will be able to figure out why –but I’m curious to know how you find the answer. When faced with problems like this, I have a favorite method – but I’m curious what other people do.

Two things to add:

1. This application, does indeed launch. When it runs, you’ll have a generic windows application, and a message box indicating success.
2. Some of you are probably thinking, “If the app doesn’t even launch, I’ll talk to the developer, since he’ll know what changed.”. But consider this scenario (which is the exact scenario that led me to create this app). You are the tester for Application Awesome. Your application is…Awesome. However, a user tells you that after installing Application Awesome, another app they love (TheApp.exe) fails to launch. WTF happened?
3. HINTS: Think for a moment about what sorts of things an application may check on launch. Could one of them be failing? How would you (could you) know?

I come across this frequently enough that I’m sure I’ve blogged about it before, but context dictates that I do it again. The story I hear goes pretty much like this:

My team really needs to improve X, but nobody is taking responsibility for it. The fix is obvious – we need our exec / manager / project owner to mandate that we do X and measure Y & Z to make sure people are doing the right thing.

Here’s a concrete example. I was on a team once where quality was in the toilet. We couldn’t even get a build to run for over a week. The solution was require everyone on the team to perform a list of actions before every check-in. Of course, the list including using flaky tools, and getting sign off on every step, even if the step wasn’t applicable to a particular check-in. Given the extra work required for check-in, developers queued up weeks worth of check-ins for one big check-in rather than go through the steps in the checklist multiple times. Day to day build quality was slightly better, but velocity was way down. While slower velocity is not necessarily a bad thing, the big problem was culture. Someone honestly thought the mandates would create a culture of quality – but the only culture they created was a culture of finding ways to skirt the damn checklist. Team morale sucked, and the product we built sucked too.

To be fair, I’ve seen some successful mandates in my career. The push for improvement in every developer to write secure software at MS worked. But that culture change was also pushed by huge monetary losses.

In general (and in every other case I can think of), mandates don’t work – especially if you are using the mandate to change culture. Yet I see mandates suggested as a solution for changing culture time and time again. I just don’t get it.

The folks at 37 Signals say it best, “You don’t create a culture. Culture Happens.”

So help let it happen. Instead of a mandate, help your team see where you want to go. Make it personal. Appeal to their own pride and values. Show them how change will help them. Find allies who think like you do. Experiment.

Let it happen. Make it happen.

Recently, I was helping another part of the team with a project. Or at least it ended up that way. There’s a particular bit of what they’re doing where I have some expertise, so I volunteered to take care of a chunk of the work where I thought I could help out.

One thing I’ve learned through experience (and through testers eyes) is to take a look at the big picture of the problem I’m solving before diving in. For example, let’s say someone asked me to change the text on a button from “Query DB” to “Query Database”. What could be a 30 second task is far from it. First, I need to make sure the button is big enough, I need to check to see if there are other buttons in the application that need similar renaming. I need to make sure the documentation (including screen shots) are updated. I probably need to make sure the localization team knows what to do with the update. Of course, I’ll see if there are any tests that look for this particular string on a button, and after I punch them in the face for testing against a hard coded UI string, I’ll make sure they fix it.

In this case, I needed to add functionality ‘A’ to a system. I know functionality ‘A’ pretty well, but in this case, in order to add ‘A’ correctly, I needed to update ‘B’ – and for ‘B’ to work, I needed to refactor huge chunks of ‘C’. I went to the team and told them that I knew what needed to be done, but it was complex (due to A, B, and C), and that while I was willing to do the work, it would take me a few days to a week to implement and test.

Then they asked me my new favorite estimation question. “How long will it take you to do the easy part.” My answer, of course**, was, “It depends. Which part is the easy part?” To be fair, they meant, how long will ‘A’ take (because they had some insight into B & C), but it was still a fun quote.

 

** Alan-ism #17: The answer to any sufficiently complex question is, “It depends”.

Joey McAllister (favorite hashtag: #expectpants) recently talked with me (electronically) about critical thinking and learning.

The interview is here in case you’re curious.

After my last post, I thought I was done commenting on stack ranking (for at least another year), but two things inspired me to write just a bit more. First, a few people flat-out asked about my opinions for alternatives; and secondly, a consultant added some ignorance to the discussion that gave me just enough motivation to write a bit more.

In this article, Dick Grote talks about the purpose of the stack rank, and for the most part, he’s spot on. Yes, a lot of companies stack rank, and yes, it’s a reasonable way in some aspects to identify both top and low performers. And then, he says this:

That’s because forced ranking makes it impossible for managers to declare that all of their employees are above average, he says, and that makes it easier to find and retain the top guns

So, according to this, one basis for stack ranking is bad managers – or lack of trust of managers. Either choice seems to me like stack ranking is going after the wrong problem. It’s not impossible to have a team of above average performers relative to other groups (statistically, of course, on a team of 11 people, five are above average, and five are below average, but all eleven may be better than fifty people on a different team. To our credit, most teams at Microsoft try to find enough peer groups so they can stack at least 50 people together, where the curve is a bit more fair.

The notion that stack ranking is a cure for bad or mistrusted managers grates on me, because I know there has to be some truth to it. Once up on a time, managers at MS had more control over budgets and rankings. The curve has always been there, but there was flexibility in rewards (and a bit more in rankings), but that’s gone.

As far as alternatives go, there is no one-size fits all model. If I got to be king for a day, I’d leave it up to organizations within the company to come up with their own methods of reviewing employees. I’d probably have to do it at the division level, but I suppose divisions could decide if they wanted flexibility at lower levels in the org. Budgets would remain the same, but it would be up to the division to decide how to distribute the rewards. The challenges of re-recruiting top talent and managing out poor performers would remain, and I’d make sure all performance plans included stipulations for both.

With my grand new scheme, Division A could decide that they wanted to continue with the traditional stack rank. Division B could decide that everyone gets the same flat bonus with extra rewards for top performers. Division C could abolish review ratings and give smaller “spot” bonuses throughout the year. Divisions D, E, and F could do something completely different if they wanted. I (as King) would offer feedback where needed, but I would trust division leaders to make decisions that would advance their business.

One of the great things about Microsoft is that you can move from division to division, completely changing what you work on, without leaving the company. An advantage of a decentralized system, is that employees could choose groups based on their review system as well as the technology or people. Some people (believe it or not!) like the stack rank review system – and those people could seek out a group that used that system. Others, may prefer collaboration above all else, and could find a group where the review model favored collaboration.

One worry of this system is what happens when someone moves from a group with one review system to a group with a different review system. For example, let’s say I’m about to get a ‘bad’ number in a group with a stack rank – so I change groups to a group with equal rewards for everyone. My simple solution (I always prefer simple) is to prorate the rewards based on time in group before the review cut off. For those that worry about dishonesty from the group I left, remember, that I always prefer trust over conspiracy, and I believe that the right thing nearly always happens in the long run.

Another worry along the same lines is how to evaluate an employee potentially coming to your group. If another divisions review system is different, how do you know how they performed. My answer to this is the same as it is today with the company wide system. Stop looking at the damn review score. Today, the ‘5’ rating is a ball and chain. You get no rewards – maybe because you’re in the wrong team, and you can’t change jobs. I don’t think we actually fire many 5’s – we just lock them into a job they don’t like until they quit…but I’m digressing… The point is that we hire smart people at Microsoft. A poor performer in one group is often a star performer in another. Talk to them, talk to co-workers and former managers and use that information to make a choice on whether they’re a fit for your group rather than a number alone.

And with that, I’m done commenting on stack rankings (besides answering questions in the comment section, of course). As I’ve said before, and as others have repeated, I don’t mind the system, and the parts I don’t like are a reasonable tax to get to play in the Microsoft sandbox.

I just wish it were better.

In my last post, I tried really hard to focus on the message of employee reviews, and how that message shouldn’t be a surprise. I said things like: “I’m not a fan of the system…”, and “It doesn’t matter how messed up the system is…” – yet the mention of the dreaded-curve of the MS review system was still the focal point of some of the comments, and of many private emails. I had always planned to follow the Three Surprises post with my view of the curve-based review system, but I’ve gone back and forth several times in the last day. If you’ve skimmed ahead, you probably figured out that I decided to go ahead and share my thoughts. There’s probably nothing new here, but I think there’s probably something here that both MS and non-MS employees will be able to learn from.

As I also mentioned in my last post, I’m not particularly fond of the curve , but consider it the tax I pay in order to get paid extremely well to do a job I enjoy. The most recent version of the review curve is the third variation I’ve experienced at Microsoft and is only 18 months old. All versions have their pros and cons, and over beers, I will gladly share the full story and dozens of anecdotes. For this forum, however, I’ll keep things (relatively) short and focus on the latest revision.

The most positive aspects about the curve are the simplicity and the transparency. There’s little mystery to the review system – peer groups (people in the same level band) are stack ranked from 1-n, and then the rank is broken into ratings groups based on known percentages. The bonus percentages for each score and level band are known, so there’s not that much mystery in the system (other than the surprises I mentioned in the last post). It’s easy to apply and easy to manage at all levels of the organization.

On the negative side (from my view at least), there are just a few things I think are worth pointing out. Microsoft values differentiation of employees. What this means in practice is that once the lines are drawn to determine review rankings, everyone in each peer group receives the exact same reward. In a ranking system of 1-5, this means that the person who just missed getting a “1” ranking gets the exact same rewards as the person who barely squeaked out of a “3” rating to get a “2”. As I mentioned above, this makes the system simple, but also can send some difficult messages. Despite this, since the goal is to differentiate employee rewards, this aspect of the curve works as promised.

It’s also a big challenge for a ‘superstar’ team. The curve applies to the team (or, for higher levels, across a division), but it’s possible to get a lower review score just by being on a good team. Even worse, the opposite is true. One can guarantee a good review score by finding a dysfunctional team and stepping in as the superstar.

And that leads to the part I really worry about – the side effects of the curve and competition. Behaviors visibly change as calibration season approaches. Some people begin to suck up in attempt to play the “review game”, while others refuse to take risks they would take at any other part of the year. A co-worker was commenting recently on strange behavior in a meeting when I asked him, “would the conversation have been different at a different time of the year?” (the answer was a resounding yes). I am a huge believer in collaboration (I once had a manager who said, collaboration was my super-power). The problem is that collaboration and competition don’t work well together. It’s in my best interest (as far as rewards go) to do everything I can to make sure all of my peers fail – and they, likewise, should feel the same. Given the system, that case could indeed happen, but fortunately most teams are able to rise above it – despite the potential hit on our own rewards and career growth. Alfie Kohn has written some fantastic books on this subject – including twoI keep on my desk this time of year: Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A’s, Praise, and Other Bribes, and No Contest: The Case Against Competition. To me, Kohn’s work says a lot about review systems in general. Combine these with Dan Pinks work on motivation, and you can form a pretty good picture for yourself with review systems.

Psychologically, as you can imagine, it’s a mess. Once the posturing is over and the calibration happens, news of the actual review scores slowly leak out – causing a zombie-like depression over a fair share of employees. After a few months, the fog shakes off, and it’s back to business as usual – for another 8-10 months or so.

I realize, that by sharing these thoughts, I’m opening the floodgates for criticism and complaints about the system. I live with this system in order to get to do what I do. If I ever get to a point where my job isn’t the best possible job I can imagine for myself, or if a review system conflicts with what I get to do, my story will change. Until then, it’s definitely a price I can pay.

I’m two months into my 18th year at Microsoft, and I still really enjoy it – most of the time at least. My job is great, I work on amazing technology, and with people smarter than you can imagine. But, for two months or so every year, the dreaded MS review monster comes out from the magic curtain of HR and freaks out employees across the company. I’m not a fan of the system, but to me, it’s the tax I pay to get to state the second sentence of this paragraph. I won’t rehash the forced curve stack rank story, but I will share one bit of advice.

It doesn’t matter how messed up the system is – review ratings should never be a surprise.

Managers may say, “but I didn’t know they were getting this rating until review time” – to which I’d say, “baloney” (actually, something stronger, but this blog is rated PG). As a manager, if you work in a stack rank system, you always need an idea of where your employees line up. When I was a manager, I made sure my team leads were stack ranked monthly. If you do it that often, it doesn’t take that long, and you can work on setting expectations throughout the year. It’s common sense, and common courtesy.

I thought I’d share (all true experiences) a few‘surprises’ from my history at Microsoft as examples. I’ve generalized the rankings for consistency where appropriate. The first surprise goes like this:

“Alex” worked his butt off all year. His team enjoyed working with him, and he finished several projects that many of his peers admired. He was quiet, but determined and proud of the work he accomplished.

At his review, he received a rating of ‘underperformed’. Alex was in shock and took the next week off to make sense of it. His manager told him that he accomplished a lot, but that he did a lot of the “wrong” work, and that his work quality was poor. This was the first time he had heard this feedback.

In this case, both Alex and his manager blew it (yes, I’m giving Alex some blame too). His manager bears the bigger portion of blame for not giving him feedback throughout the year, but Alex needed to ‘manage up’ as well and ask his boss for feedback. Alex left the company soon after and his now near the executive level at another software company.

“Beth” was almost a celebrity on her team. She had a lot of visibility around the company, and her team was excited to have her as a team member. She was one of those people who made everyone around her a strong performer, and her teammates heaped praise on her as part of the review process. She put a ton of effort into crafting performance goals that would show her value on the team, and even presented them to the Vice President of her division for feedback and approval. Her manager gave her feedback throughout the year that she was doing well…and then she got an ‘underperformed’ on her review. Both Beth and her manager were shocked! You see, at Beth’s level, her skip level manager expected something different from her. There was no warning or feedback before this.

Beth quit the team immediately and found another job at Microsoft.

In this case, I put the majority of the blame on Beth’s manager (for not looping in his manager earlier), but one thing I’ve learned at MS, at least, is that you always want to put at least a little effort into knowing your management chain and knowing what they expect of you.

The last surprise is a first hand experience – it’s the only time I’ve ever surprised an employee at review time. At this time, we had a review system that went in half-point increments from 2.5-5.0. Most employees received 3.0 or 3.5 ratings. A 2.5 meant your job was at risk, and 4.5 and 5.0 ratings were reserved for (roughly) the top 1% and .01% respectively. The conversation went like this…

“Evan – you’ve done a great job this year. You’ve built a great team, done some fantastic technical work, and helped build a strategy for the team…but as I look at what your peers have done, and what the rating systems dictate, I can’t justify giving you a 4.0 rating…

<I watch Evan’s eyes as they go from confused, to hurt…and then, just has his eyes get a twinge of Hulk rage in them, I continue…>

So, based on what you accomplished and how you got it done, I have no option other than to give you a 4.5 rating.

Yes – it was a bit mean, but given the nature of the surprise, Evan was extremely happy. He’s still at MS, and has grown into a major leader in his division.

As far as I’m concerned, that’s the only type of surprise that should ever happen with an employee review (just be careful of the timing on your delivery :} ).

As I write this, I’m waiting (literally, waiting on hold) to give a webinar for Swiss Testing Night. It’s a twenty minute presentation – which I love (see my last post for another twenty minute presentation from me. I would love to see a test conference filled with nothing but 20-30 minute presentations someday (and would probably even help organize if someone were willing to really do it).

But this post isn’t about conferences, it’s about learning. These days, I’m a bit obsessed with ideas and learning, and can’t help look for patterns of learning in nearly everything I do. It’s intern season at Microsoft, and we have a few dozen eager young faces floating around the hallways making great things happen. As part of their experience here, the interns are invited to a weekly lecture series on a variety of topics – including a talk from me later this summer. I haven’t settled on a topic yet, but I expect it will be something along the lines of how what you learn at university probably won’t help you in the real world – the point being that the bits of facts and knowledge will help, but knowing how to learn is the most critical knowledge one can gain at university. I’ll find some ways to make sure the message works without sounding too much like a crazy old man, but we’ll see what happens.

Yesterday, I had a great visit with a computer science student at a university in Georgia, and I had a wonderful conversation on this same topic I’m on the alumni board for the school of Arts and Humanities at my alma mater (Central Washington University). Last  week, the chair of the music department emailed me (we’ve met once before) and asked me if I could talk to his son (the CS major mentioned at the beginning of this paragraph). They came over yesterday for a quick visit and chat. Over lunch, and during our tour, we talked a lot about the power of learning, systems thinking, and critical thinking. I’ve interviewed MIT grads who could practically recite a textbook, but couldn’t think – so it was great to talk to this kid studying computer science at a liberal arts school who seemed to already have a grasp of how he fits into the real-world.

I’ve said this before, but it’s worth stating again. Programming is easy – especially with the languages available today. What’s really hard is implementing the right program, or writing code to solve the problem in the right way. This is true for applications as much as for test automation – and you need to use your brain and learning skills to have a chance for success.