Windows 95 Nostalgia

I feel like today’s a good day to share a few stories about my first few months at Microsoft, and the (very) small part I played in shipping Windows 95.

My start at Microsoft is a story on its own, and probably worth recapping here in an abbreviated form. I started at Microsoft in January 1995 as a contractor testing networking components  for the Japanese , Chinese, and Korean versions of Windows 95. I knew some programming and even a bit of Japanese (I later became almost proficient, but have forgotten a lot of it now). I also knew, for better or for worse, a lot about Netware and about hardware troubleshooting, and that got me in the door (and got me hired full time 5 months later).

Other than confirming that mainline functionality (including upgrade paths) were correct, there were two big parts of my job that were unique to testing CKJ (Chinese, Korean, Japanese) versions of Windows. The first was that at the time, there were a dozen or so LAN cards (this was long before networking was integrated onto a motherboard) that were unique to Japan, and I was (solely) responsible for ensuring these cards worked across a variety of scenarios (upgrades from Windows, upgrades from LanMan, clean installs, NetWare support, protocol support, etc.). One interesting anecdote from this work was that I found that one of the cards had a bug in its configuration file causing it to not work in one of the upgrade scenarios. Given the time it typically took to go to the manufacturer to make a fix and get it back we decided to make the fix on our end. Because I knew the fix (a one liner), I made the change, checked it in, and that one liner became the first line of “code” I wrote for a shipping product at Microsoft.

The other interesting part of testing CKJ Windows was that Windows 95 was not Unicode; it was a mixed byte system where some (most) characters were made up of two bytes. Each language had a reserved set of bytes specified as Lead Bytes, that indicated that that byte, along with the subsequent byte were part of a single double-byte character. Programs that parsed strings had to parse the string using functions aware of this mechanism, or they would fail. Often, we found UI where we could put the cursor in the middle of a character. The interesting twist for networking was that the second byte could be 0x7c (‘|’), or 0x5c (‘\’). As you can imagine, these characters caused a lot of havoc when used in computer names, network shares, paths, and files, and I found many bugs testing with these characters (more explanation on double-byte characters, along with one of my favorite related bugs is described here).

While I didn’t do nearly as much for the product as many people on the team who had worked on the product for years, I think I made an impact, and I learned so many things and learned from so many different people.

“Why the UI?”

Readers of my blog know my stance on UI automation. But, as I’ve forgotten my StickyMinds password, and the answer is longer than 140 characters, so I’m responding here.

This article from Justin Rohrman talks about the coolness of Selenium for UI testing. In a paragraph called, “Why the UI”, Justin wrote:

The API and everything below that will give you a feel for code quality and some basic functionality. Testing the UI will help you know things from a different perspective: the user’s.

I like everything else in the article, but that second sentence kills me. Writing automated tests for the UI is as close to a user perspective as I am to the moon (I’m only on the 20th floor). I’m going to do Justin a favor and rewrite that paragraph for him here. Justin – if you read this, feel free to copy and paste the edit.

…some basic functionality. Testing the UI is difficult and prone to error, and automation can never, ever in a million years replace, replicate, or mimic a real users interaction with the software. However, sometimes it’s convenient – and often necessary to write UI automation for web pages, and in cases where that happens, Selenium is obvious choice.

Justin – your work is good – I just disagree (a LOT) with the trailing sentence of the paragraph in question.

Back to work for me…

Building Quality

I mentioned in my last post that I have a new job at Microsoft (and I discussed it a bit more on the last AB Testing). During the interviews for the job, I talked a lot about quality. I used the agile quadrants as one example of how a team builds quality software (including my roles in each of the quadrants), but I also talked about quality software coming from a pyramid of activities and processes. I’ve been dwelling on the model for the last week or so, and wanted to share it for comments and feedback…or to just brain-dump the idea.

Processes / Practice / Culture

The base of software quality (and my pyramid) is in the craftsmanship and approach of the team. Do they care about good unit testing and code reviews, or do they check in code willy nilly? Do they take pride in having a working product every day, or does the build fall on the floor for days on end? The base of the pyramid is critical for making quality software – but on established teams can be the most difficult thing to change.

Code Quality (correctness)

An extension of PP&C above is code correctness. This is a more granular look specifically at code quality and code correctness. This includes attention to architecture, code design, use of analysis tools, programming tools, and overall attention to detail on writing great code.

Functional Quality

Unit tests, functional tests, integration  / acceptance tests, etc. are all part of product quality. I italicize, because for some reason, some folks think that quality ends here – that if the tests pass, the product is ready for consumers. (un?)Fortunately, readersimage of this blog know better, so I’ll save the soapbox rant for another day. However, a robust and trustworthy set of tests that range from the unit level to integration and acceptance tests is a critical part of building software quality.

Automate Everything

There are some folks in software in the “Automate Everything” camp. A lot of testers don’t like this camp, because they think it will take away their job. Whatever.

As far as I can tell from my limited research on this camp, Automate Everything means automate all of the unit functional and integration tests…and maybe a chunk of the performance and reliability tests. For some definitions of “Everything”, I agree. Absolutely automate all of this stuff, and let (make) the developer of the code under test do it. The testers’ mind is much better put to use higher up the pyramid.


Performance, reliability, usability, I18N, and other non-functional requirements / ilities are what begins to take your product from something that is functionally correct to something that people may just want to use. Often, the ilities are ignored or postponed until late in the product cycle, but good software teams will pay a lot of attention to this part of the pyramid throughout the product cycle.

Customer Quality

It doesn’t matter how much you kick ass everywhere else in the pyramid. If the customers don’t like your product, you made a shitty product. It may be a functionally correct masterpiece that passes every test you wrote, but it doesn’t matter if it doesn’t provide value for your customers. Team members can “act like the customer”, be an “advocate for the customer”, or flat out, “be the customer”, but I’ll tell you (for likely the twentieth time on this blog), as a member of the product team, you are not the customer! That said, this is the part of the pyramid where good testers can shine in finding the fit and finish bugs that cause a lot of software to die the death of a thousand paper cuts.

Now, if you do everything else in the pyramid well, you have a better shot at getting lucky at the top, but your best shot at creating a product that customers like crave is to get quantitative and qualitative feedback directly from your users. Use data collection to discover how they’re using the product and what errors they’re seeing, ask them questions (in person, or via surveys), monitor twitter, forums, uservoice, etc. to see what’s working (and not working), and use the feedback to adapt your product. Get it in their hands, listen to them, and make it better.

More to come as I continue to ponder.

Twenty Years…and Change

In January of 1995, I began some contract work (testing networking) on the Windows 95 team at Microsoft. Apparently, my work was appreciated, because in late May, I was offered a full time position on the team.

My first official day as a full time Microsoft employee was June 5, 1995.

That was twenty years ago today!

I never (ever!) thought I would be at any company this long. I thought computer software would be a fun thing to do “for a while” – but I didn’t realize how much I’d enjoy creating software, and dealing with all of the technical and non-technical things aspects that come with it. I learned a lot – and even though my fiftieth birthday is close enough to see, I’m still learning, and still having fun – and that’s a good thing to have in a job.

I’ve had fourteen managers, and seventeen separate offices. I’ve made stuff work (and screwed stuff up) across a whole bunch of products. I’ve done a ton of testing, entered thousands of bugs, and written code that’s shipped in Windows, Xbox, and more (not bad for a music major who stumbled into software).

In a nice bit of coincidence, my twenty-year mark also is a time of change for me. After two years working on Project Astoria (look it up – it’s really cool stuff), it’s time for me to do something new at Microsoft…something that aligns more with my passions, skills, and experiences – and something that shows what someone with over two decades of software testing experience can do for modern software engineering.

I’ve joined (yet another) v1 product team at Microsoft. Other than a few contract vendors, the team of a hundred or so has no software testers. They hired me to be “the quality guy”. This set up could be bad news in many worlds, but my job is definitely not to try to test everything. Instead, my job is to offer quality assistance, help build a quality culture, assist in exploratory testing, and look at quality holistically across the product. I don’t know if any jobs like this exist elsewhere (inside or outside of Microsoft), but I’m excited (and a bit scared) of the challenge.

More to come as I figure out what I do, and what it means for me as well as everyone else interested in software quality.

<sigh> Automation…again

I think this is the first time I’ve blogged about automation since writing…or, to be fair, compiling The A Word.

But yet again, I see questions among testers about the value of automation and whether it will replace testers, etc.. For example, this post from Josh Grant asks whether there are similarities between automated trucking and automated testing. Of course, I think most testers will go on (and on) about how much brainpower and critical thinking software testing needs, and how test automation can never replace “real testing”. They’re right, of course, but there’s more to the story.

Software testing isn’t at all unique among professions requiring brain power, creativity, or critical thinking. I challenged  you to bingoogle “Knowledge Work” or Knowledge Worker”, and not see the parallels to software testing in other professions. You know what? Some legal practices can be replaced by automation or by low-cost outsourcing – yet I couldn’t find any articles, blogs, or anything else from lawyers complaining about automation or outsourcing taking away their jobs (disclaimer – I only looked at the first two pages of results on simple searches). Apparently, however, there are “managers” (1000’s of them if I’m extrapolating correctly) who claim that test automation is a process for replacing human testers. Apparently, these managers don’t spend any time on the internet, because I could only find second hand confirmation of their existence.

At risk of repeating myself (or re-repeating myself…) you should automate the stuff that humans don’t want (or shouldn’t have) to do. Automate the process of adding and deleting 100,000 records; but use your brain to walk through a user workflow. Stop worrying about automation as a replacement for testing, but don’t’ ignore the value it gives you for accomplishing the complex and mundane.

Forwards and Backwards

What a year it’s been so far. I’ve been away from blogging, and I’m not quite sure if I’m back yet, but I expect so…and here’s why.

The project I’m working on at Microsoft is no longer a secret. I’ve never blogged a lot about product specifics, but since a big chunk of my work for the last year has been way off my normal path (linux, java, etc.), I just didn’t find a lot to share (not necessarily because I wanted to help keep the project under wraps, but because pushing through n00b questions on tools and language constructs probably isn’t very exciting for my typical audience).

There’s also been a lot of change – net little change in what I do day to day. For the last several years, my role (which many wouldn’t call testing) has been to figure out what’s not getting done, and make sure it gets done. With Xbox (and Lync before that), that meant helping the test team – either by coaching or mentoring, or developing strategy, or building tools, or asking questions about the product, or whatever needed to be done – by myself, or through others.

That (figuring out what needs to get done) is exactly what I do for my current team. But on this team, those holes and gaps have been mostly about improving our builds, CI, testing, and other parts of our systems that make our build->measure->learn loop more efficient and effective. I help with testing too, but in different ways than in the past.

I’m going to give a keynote at STAR Canada next month where I’ll talk a bit about what I do, and how it’s changed over the last few years. I’ll also try to dispel the silly myths  / bad interpretations that MS doesn’t do testing anymore. As I develop that talk, I expect I’ll come up with some good blog fodder to share and to help develop ideas. Crap – now that I think about it, I should have that talk a lot further along than it is right now. I better put some time on that right now.

If you haven’t already, please check out the AB Testing podcast.

Bibliographies, Books and Stuff

A while back (no link cause I’m lazy), I mentioned that one thing (among many) missing from How We Test SW at MS was a bibliography of testing books. While we didn’t refer directly to any books when writing the book, the influence of several testing books is definitely there, and worth noting for those who want to study where the craft has come from (which is one way to better understand where it’s going). This post will now serve as the official bibliography of the 2008 edition of How We Test Software at Microsoft. That should get at least one nagger off my back.

But First…

My bookshelf is filled with four different types of books (technically, three – one type is a subset of another.

  1. Business-ish books. I have (and read) a lot of books on leadership, innovation, etc.. I think these are important for all leaders and those who want to influence (definitely not just managers). These books contain the most read and most loved books in my collection – but none of them are included in this list.
  2. Books on thinking. I also read a lot of books about thinking and motivation. Examples include Michalko’s Cracking Creativity and Thinkertoys, Kahneman’s Thinking Fast and Slow, Pink’s Drive, and A Whole New Mind, Weinberg’s General Systems Thinking, and Chabris’s The Invisible Gorilla . I think books like these are a great resource for anyone who does knowledge work. A good way to push my buttons (as many of you know) is to take a nugget from a book like these (the gorilla is, of course, my favorite), and announce to the world, “OMG!!! THIS IZ JUSST LIK TESTING!!!!!!!!” These books aren’t in this blog-iography either.
  3. Software books. The final category is books about software engineering. Note that there are examples of software books that overlap into the above categories.

Please don’t ask where book “Foo” fits into the above categories. The category list above is arbitrary and I just made it up. I may change it later (or by the time I post this).

Category #3 above also includes books on software testing – and after an introduction that is now likely longer than I expect the list of testing books and notes to be, let’s get started. Below is a list of the software testing books that I believe have contributed in some way to my test education (and hence, my own teaching and the writing of HWTSAM), as well as directly and indirectly to many of my peers. In no particular order (other than the order you can find them on the shelf in my office from right to left):

  1. Software Testing Techniques by Boris Beizer. This book has a comprehensive overview of a whole lot of testing techniques. Some of them are good, some don’t apply, and some are complete bullshit. I tell a story about my journey with software testing books where I say, “When I read my first book on software testing, I thought I learned it all. After reading a second book on testing, I was confused. After reading a third testing book, I began to form my own opinions. For me, I don’t think it’s fair to determine where formal models apply and don’t apply until I really understood them – and this book gave me that background.
  2. Software Testing: A Craftsman’s Approach, by Paul Jorgensen. This is another book in the “comprehensive” category, and deals entirely with software testing techniques. Again, it’s an exercise for the reader to figure out the best ways and contexts to apply the techniques (and to filter out the bullshit), but as far as basics of testing goes, it’s one of my favorites. I know that Bj was influenced heavily by Jorgensen’s work, and I think it shows in his chapters on functional techniques.
  3.  A Practitioner’s Guide to Software Test Design, by Lee Copeland. I hate to say this, because Lee occasionally reads this blog, but this is my absolute favorite book on the basics of Test Design. For me, it was so much easier to read than the Jorgensen and Beizer books (and the even Meyers book I’m about to mention), and I think it’s much easier for the reader to take the information in the book and apply it to their own work. I don’t recommend reading just one book on test design (see my comments on the Beizer book), but this is one I think is worth having on your shelf.
  4. imageThe Art of Software Testing by Glenford Myers. This one is sort of the classic of software testing. It too, focuses on functional techniques (and introduces the classic triangle testing problem). Note that this book Ii s generally way overpriced ($130 on Amazon right now)– look for a used copy if you want to own it.
  5. Testing Object-Oriented Systems, by Bob Binder. I probably  mentioned this book in the chapter on model-based testing. This is the book where I first studied MBT, and also the book that got me thinking deeply about test patterns and patterns in general (I was inspired by this book to finally read the GOF book, as well as Alexander). Note that this book can double as a weapon. It’s just under a zillion pages long, so I recommend reading it in chunks rather than in a single three-week sitting.
  6. imageSoftware Testing Techniques by Loveland and Miller. In a way, this book could be called How We Test Software at IBM – or, perhaps, Things you can learn from Mainframe Testers. This book came out just a few years before we began writing HWTSAM, and probably subconsciously influenced my decision to agree to think about possibly writing a book about testing at Microsoft. There are a lot of lessons to be learned from this book, and although it’s applicability to my world of testing has waned quite a bit in the last decade, there are definitely some nuggets here for anyone.
  7. This is the placeholder for something obvious I’ve missed. I’ve read dozens of other books on software testing, and unless someone has borrowed something significant without asking, I have to say that those are the most likely books that influenced my world view of testing – and likely what we drew from (at least indirectly) when writing HWTSAM.
    Well… those six books and a combined half a century of practical experience from the authors, and a millennia of experience from the combined contributors.

My year of working sparsely

Warning, non-software post follows.

2014 was an interesting year for me. The year began with the trailing two weeks of a four week visit to Australia. Then, I spent July and August in France, then took another week off when I returned to take care of my dad. Finally, I took the last two weeks of the year off to be with family. With other short vacation days thrown in, I took nearly 13 weeks off in 2014 (if my math is right, that’s a quarter of a year). Most of that time off came from an award I received several years ago that allows me a one-time eight-week vacation (a sabbatical of sorts), and I also used just about all of the five weeks of regular vacation time that Microsoft gives me.

I mentioned the above (in much shorter form) on twitter this week. In the discussion, I mentioned that I’ve been at MS for nearly 19+ years, and it’s a once in a career sort of thing, and Geir Gulbrandsen asked if I’d do an encore after an another 19 years. I answered that I’d likely be in some stage of retirement by then, and that led to a discussion on what retirement is…and it made me want to dump my thoughts into a form longer than 140 characters.

So here goes.

A lot of twitter folks said things like, “If you do what you love, do it forever” or, “I’ll never stop working if I enjoy the job”. I agree with these folks. Even in “retirement”, my dad worked for years just to keep from getting bored. I get bored easily too, and I really like my field of work, so why would I even think about retirement (in any form)?

Because I like things other than working too. I like to travel (actually, I don’t like the travelling part – I like being in other countries). Spending two months in a foreign country (or a month each in two different countries) isn’t something someone can do often when working a full time job. Or, I may write another book someday – but I know from past experience that I didn’t like the schedule I had to keep to work full time while writing. I may take a class. I may want to work in a winery. Who knows. All I know is that at some point in my life (and probably within 10 years or so), I’ll want to work much less than I’m working now. Somewhere in the twitter stream, I said (tweeted) this:

I’ve known that I wanted “retirement” options from the time I entered the workforce (as a middle school music teacher when I was 22 years old!). I’ve saved aggressively – maxing out my pre-tax 401k for the last 19 years, and contributing as much as I can afford to post-tax 401k over the past several years. I feed my IRA every year and save whatever else I can. The only debt I’ve had for the last two decades is a mortgage. And, although we went a little crazy with vacations over the last year, we travel cheaply, and save as much as we can while still enjoying life. To be fair, I’ve been successful working for a large company, so my income level enables me to save more than a lot of people, but the point is that I’ve tried to save what I can, so at some point, I have a choice to work – or not, as I choose.

Any sort of retirement for me is still quite a few years off. My kids are in school for another eight years, and will probably go to college after that (I started 529 plans for both of them when they were born, and I expect I’ll have enough saved to cover a large portion of costs for all but the most elite schools). The point is that I think a lot about what I need to do to get myself to a point where I can cover most or all of my expenses without needing to work.

But I’ll still work when I want to.