stuff you should know
I uploaded my Eurostar keynote slides to SlideShare. As with most of my talks, the slides don’t read well on their own (I sort of hate bullet points). I rarely repeat talks, but given timing and audience, I may consider giving this again as a webinar at some point (I’m also done travelling for a while).
Happy to answer questions if they come up.
I’ve been home for a few days from by brief trip to Amsterdam for EuroStar. As I mentioned in a previous blog post (or two), I used my Surface RT as my only device for the trip. Overall, it went quite well. I was able to accomplish most of the things I wanted to, including using my RT to drive my presentation. No glitches in the OS, and no glitches in PowerPoint. Some of my highlights and lowlights include:
Highlights
Lowlights
The Conference
This was my first EuroStar, and I had a fantastic time. The conference begins with a day and a half of tutorials, followed by two and a half days of talks. My talk, “Test Innovation for Everyone” was the opening keynote. I had a good time with the talk, and received some good feedback. . If I ever give the talk again (something I rarely do), I’d clean those bits up a bit. There are a few places where I could have stitched together things a bit smoother, but overall I was happy with it.
Esther Gons drew a picture of my presentation that sums up my topic pretty well.
I also met a lot of really cool people – many who I knew of from blogs or twitter, but had never met in person. I also met some folks from Skype there who all seemed pretty sharp. I would try and list the names, but there are way many to list, and honestly, if you saw the list of testing celebs I got to hang out with it would just seem like name dropping anyway.
I had a great time, and hope I get a chance to see Europe and some of the great testers there soon.
If you read my last post (sorry, no link – I’m not ready to try that yet), you’ll know that I’m attempting to use my Microsoft surface device as my only electronic device on my trip to Amsterdam for Eurostar.
So far, it’s working ok. The cover/keyboard works better than I though tit would – although I seem to have a problem with it registering the space bar a lot. You really do need to be a good touch typer to have confidence with this thing.
My presentation tweaks aren’t quite done, so I’ll have to put this thing through a few more paces to see what I really think. I’m also curious to see how limited the Surface is with no internet connection (I assume I’ll find myself in that situation on this trip, but we’ll see).
Boarding in a few minutes – more updates from Amsterdam.
ugh – first problem is that the wordpress app won’t publish my post…
Continued: Because I can’t post, I’ll continue with my update. My first bad experience has little to do with the surface – it’s that the wordpress app is lame. I won’t describe the depths of the lameness, but I will say that I’ll likely post this directly from my blog rather than the WP app.
I just spent 30 minutes or so tweaking my slides – that was a good experience. Typing is still a bit weird (and I still miss the space bar about every ten words or so), but for formatting, moving slides around, etc. the Surface works remarkably well. I find the screen size and resolution seem to be just about perfect for working on an airplane (I’ll find out if I say the same for hotel rooms and coffee shops in an upcoming post).
More later (when I find Wi-Fi).
I had a not-so-good week last week.Lot’s of stuff too private for blog sharing, but let’s just say it was a cascade of dark and bad times that nobody should have to go through. One of the bits of bad luck was that the LCD on my laptop broke. Normally, this wouldn’t be so bad. Most of the time (i.e. unless I’m traveling), I log into my laptop via terminal server, so a broken screen has no impact. This is my “work” laptop, and it’s overdue for an upgrade, so normally, everything would line up well.
This is not a normal week. I leave for Amsterdam to speak at Euro STAR on Sunday, and there’s no way to get a replacement laptop through work on time. I looked into getting the screen replaced, but at $500, I didn’t feel like it was cost effective – I thought the same of renting a computer for $120 for the week. I have a strong network at MS, and pinged some of my colleagues on the Windows team for spare hardware, but that well was dry too. I begged admins, IT, and anyone I could find to let me use a laptop for the week, but everything fell through.
So, I figured it was time to buck up and buy a personal laptop. Microsoft is giving me (and every employee) a Surface RT, but we won’t get those until January. I looked around a bit, and felt a little strange paying (at least) $700 for a laptop that I probably wouldn’t use much after this trip (have you figured out yet that I hate spending money?).
In the end, I decided to go all in – to jump on the Windows Surface bandwagon and buy a Surface RT (yes – the same device I’ll get for from MS in a few months). If I like it, it will be nice to have two (if I love it, I’ll give one away!) The challenge I’ve given myself is to see how productive I can be using the Surface on the plane and in my hotel room – and using the Surface as my powerpoint source for my keynote at the conference. It’s going to be a fun few days, and I expect I’ll blog a bit about how it’s going along the way. I only hope that if my experience goes south that the Windows bigwigs aren’t going to get mad.
We’ll see what happens.
Some news about my involvement in upcoming conferences I’m probably overdue to share.
First of all, I’ll be giving a keynote at Euro STAR on November 6th. This is my first trip to Euro STAR, and my first visit to Amsterdam (at least my first visit where I’ll leave the airport). There are a lot of talks I plan to attend, but I’m a bit bummed that I’ll miss the last day of the conference, and will miss a talk by Alexandra Schladebeck on What Agile Teams Can Learn From World of Warcraft. I’ve used this metaphor many times over the years, and I’m excited to see that it’s not just me that sees the connection.
Beyond that, I’m done with conferences for the next eleven months or so while I go back to paying full attention to my family and my day job (but I may give a webinar or two this winter given sufficient interest).
I’m fortunate to have a team and manager who support me attending a few conferences a year, but while they’d probably be supportive of me agreeing to attend more conferences, my day job is much cooler than most, so I’m cutting way back on my travel so I can do even more cool stuff. I have made a commitment to attend STAR West next fall, but other than that, I’m consciously making no commitments (although I would attend a good peer conference if one came up).
Finally- I want to mention that Swiss Testing Day – one of the best conferences I’ve ever attended (both in organization and in content) is coming up. I’m not attending this year, but I do encourage anyone who is, or will be near Zurich on March 13, 2013 to consider attending – I am confident you won’t be disappointed.
On twitter, @SeanNoxious asked me about my most memorable bug. The answer is way too long for twitter, so I thought I’d document one of my favorites here.
When I first joined Microsoft way back in 1995, I was a tester for networking components on Windows 95. One of the areas I owned was testing networking on Japanese, Chinese, and Korean versions of Windows.The early versions of windows and all of the 9x versions of windows didn’t have Unicode. Character sets were multi-byte on those systems, meaning that characters may be made up of one or two bytes. Alpha-numeric characters were usually single-byte (wide / two byte versions also existed (when a character had both a single-byte and double-byte version, the single-byte version was referred to as the half-width character. There’s a lot more to say about double-byte characters, but I’ll skip ahead to the good part.
The way windows could tell the difference between a double-byte character and a single-byte character was the lead byte. A certain range of characters are designated lead bytes. They always indicate that the lead byte, and the next character in the stream are combined to make a double-byte character. The Japanese character for ten (juu) has a hex value of 0x8F5C. The 8F indicates that the following characters are part of a double-byte character. The interesting thing about this particular character (and all characters with an 5C trail byte) is that 5C is the hex value for a backslash – a character very interesting for network testing on Windows.
There were other characters with interesting trail bytes, but 5C characters were predominant in my testing, and I found a lot of bugs. One day, I was testing copying long files with 5C trail bytes to a Japanese version of Windows NT 3.51 (newly released!), and my test hung. I figured someone else was using the server and rebooted it or was debugging it for some reason, so I walked down to the lab to check it out; and it was sitting at a blue screen. I stared for a minute before doing what every tester does. I rebooted it, and ran back to my office to see if I could do it again.
Sure enough, I had found a serious bug in the NT networking components. One that I knew was tested by people way more experienced than I was, and I was pretty excited.We found other bugs in NT, and plenty of multi-byte bugs in networking, but crashing a remote server has to be one of my favorite finds.
I have another cool backslash story, but I’ll save it for another post.
I spoke recently on the topic of “Customer Focused Test Design” (synopsis: customers don’t really care how much functional testing testers do, or how many bugs they find. A test approach that favors scenarios, -ilities, and customer feedback is better; and a bunch of examples for emphasis/proof).
Part of the approach I suggest (and have had some success with), is doing testing of performance, reliability, security, privacy, world readiness, and usability (including accessibility) early in the product cycle. Early, as in don’t bother with functional testing, do the ilities instead. The premise (and my experience) is that testing ilities tests the underlying functionality by default, and that programmers are (in general) doing a better job testing for functionality during code development. For more details, I’ll probably have to revive the whole talk – and that’s not what this post is (intended to be) about.
There’s one –ility that is critical to the customer perception of quality, but it’s not quite like the others – supportability. Customers hate problems, but they love finding solutions to those problems. Online forums, customer connection programs, twitter, facebook, and other online platforms are quickly becoming support forums for a lot of software. Software companies who engage with customers actively are creating happy customers.
Zappos has created a fantastic culture of customer service and have won customers for life (I’m one of them) by actually caring about their customers.
Recently, I moaned on twitter that I couldn’t get my (aisle) seats confirmed for a long flight from Australia to Seattle on a flight booked through Orbitz. Within minutes, @OrbitzCareTeam contacted me on twitter and booked and confirmed my seats. I fly a lot (too much), and I can book my flight with any discount airline– but Orbitz will get my business.
Last month, I visited an Xbox call center, talked to some of the employees there, and listened in on some calls. I was ecstatic to see that nobody works from a script and there are no quotas or other incentives to push people through the system. I listened to several calls where the support folks took their time helping people solve problems step by step (on occasion, converting customer rage into customer appreciativeness). Most interesting was that none of the calls I listened to had anything to do with the Xbox console. Our support people answered questions about entering codes for other manufacturers games, helped customers reset live id passwords, and a variety of other topics related to – but not part of the actual console. Every customer hung up happy – I was blown away.
If you want to be a successful software company, you have to care about customers. In addition to keeping them happy once they have used your software or service, you need to respond to their needs and give them what they need. In The Lean Startup, Ries drives home the point of iteration as a means to obtain validated learning; with the premise that any work that does not provide value to the customer to be wasteful. In other words, listen and learn – often.
Sometimes companies get it wrong. I recently had an issue with an application that was overlaying a decoration on my explorer icons. I found the overlays to be distracting and a detriment to my productivity. When I searched online for a workaround or solution (or sympathy), a representative of the company had this to say (bold mine).
Our thinking was that we want the app to be running in the background, all the time. We want to to blend into your experience so you almost never have to interact with [the application] to check the status of things. So we figured icon overlays were a subtle way to do this, while reassuring people that the app was indeed running.
Maybe we can add an option in the future to toggle the overlays, but honestly I wouldn’t hold your breath. There’s plenty of other cool stuff we want to add first 🙂
Note the pronouns in bold. Our thinking…We want…we figured…we want… – these are the statements of Engineering Focused Engineering (note the intentional idiocy of that phrase)
To be clear, I like this product. And on my own, I figured out how to disable the overlays. I also don’t know if the statements above reflect the opinions of one person, or a whole software team…
But the statements above are the statements of an organization that doesn’t give a $#@t about their customers. What’s worse, is that this software’s main competition is an application that is praised for its customer focus. In my opinion, it’s unprofessional to approach software development in this way – unless of course you don’t actually want to have customers use your software.
Isn’t it about time to put customers first? Always? I’m all for shifting test design to favor scenarios customers care about, but successful software projects are going to require a customer focus for the lifetime of the product – from concept to development to deployment and beyond.
It’s time for the rise of the customer.
Note: no direct information on software testing or software engineering is in this post. This is purely a trip report (that happens to be related to a trip I took to talk about testing).
As I begin writing this, I’m in the middle of a trip to New Zealand and Australia. The main purpose of the trip was a bit of a speaking tour for SoftEd (for the STANZ and Fusion conferences). I gave a keynote and a workshop in Christchurch, NZ on Monday – repeated the same thing on Tuesday in Auckland, and then hopped across the ditch to Sydney to give a talk on Thursday and a workshop on Friday.
Although this trip marks my first visit to Australia, I spent three weeks in New Zealand almost exactly ten years ago. I arrived Saturday morning so I’d have a bit extra time to acclimate to the new time zone, but I didn’t really need it. Through a combination of horrible movie selections and three seats to myself, I managed to sleep quite a bit – and apparently just the right amount so that morning in Christchurch felt a lot like morning in Seattle. The hotel was nice – as far as hotels based on medieval themes go (this being my first medieval themed hotel, I was quite happy). I went for a run, showered, explored a bit, and then went in search of dinner. Although I spent a few days in Christchurch ten years ago, I wasn’t sure how much I’d remember. I also wasn’t prepared for just how devastating the earthquake in February, 2011 was. Nearly two years later, the bulk of the downtown area is still fenced off and unsafe. I found the hotel where I stayed in 2002 (closed), and a restaurant I remember (also closed). The weird part was how dead the downtown area was on a Saturday night. While I wandered around pondering the destruction, I suddenly realized that it was silent. Other than the flap of a few flags and banners in the strong Christchurch wind, there was nothing – complete stillness in an area that was bustling nearly 24-hours a day during my last trip. I eventually found my way back to the area near the conference hotel – apparently where at least some of the liveliness had moved, and found some food.
I was up early Sunday morning and went for a long run in the park. I met up with fellow conference speaker Elisabeth Hendrickson for a coffee run that evolved into a wonderful walking tour of the area surrounding the hotel. We managed to invite ourselves to a tour of a nature walk and had some great conversations about software engineering, testing, and our shared love of walking miles and miles in foreign cities. Although I’ve met Elisabeth before, one of the big benefits of this trip was getting a chance to get to know her better. She has a sensible holistic approach to software engineering that I really like (and unfortunately, don’t see nearly enough of among people in the testing community).
Monday was a blur – I think I went for a run in the morning, gave a keynote later in the morning, and delivered a workshop in the afternoon. Then we all flew to Auckland, where we checked in, and I decided I preferred sleep much more than dinner.
Tuesday was more of the same. I changed my talk a bit – I changed the workshop a bit. Not only do I not like giving the same talk twice – I think I’m incapable of giving the same talk twice. I suppose this keeps things fresh (for me, at least). After the conference a group of us had a wonderful fish dinner at a nearby restaurant.
My flight on Wednesday wasn’t until 6:00pm, so I used the morning and afternoon to explore Auckland. I spent quite a bit of time in Auckland (3-4 days) in 2002, and was eager to re-explore. It’s funny how we remember things. There were restaurants and points of interest I remembered quite well (and went in search of a few of them), but it was a bit of a mind trip to notice places and things that I didn’t remember until I saw them (“Oh wow – I ate at this exact restaurant ten years ago…”)
I made a brief stop at the Microsoft Auckland office during my wandering. I knew I wanted to go to Melbourne while in Australia, but I hadn’t bothered to book a flight or lodging yet. Nor did I have any lodging in Sydney after the conference. I took advantage of some peace and quiet and (apparently rare) high-speed internet and got all of my bookings squared away.
The Fusion conference was Thursday and Friday. The opening keynote was Emma Langman, and I really enjoyed her talks (she also gave the closing keynote on Friday). Emma is a Change Magician (cool title, huh?), and knows a ton about organizational change and systems thinking. I applaud SoftEd for inviting her, and think she did a great job capturing the essence of the conference and coming up with relevant topics for the audience.
On top of that, I was a bit intimidated that she attended my Thursday session (on customer-focused test design), but managed to keep it together as I babbled for an hour about test design.
Thursday night, we had a dinner keynote from Dr. Peter Ellyard (Australia’s most prominent futurist). I spoke to him for a few minutes before dinner, where he shared this quote that stuck with me:
Vision without strategy is a daydream. Strategy without vision is a nightmare.
(note that this is a variation of a Japanese proverb that replaces the word ‘strategy’ with ‘action’)
After dinner a few of us hung out and had a few more drinks. I have a vague memory of late-night Karaoke, but the details of the night may be forever lost.
Friday morning came quickly, and I gave my last talk of the conference. All-in-all, it was a fantastic experience, and I am grateful to SoftEd for the opportunity.
Saturday morning, Elisabeth Hendrickson and I met up for a sequel walk and talk event around Sydney, culminating with one of the best cups of coffee I ever had (along with a delicious salmon sandwich). I spent the rest of the weekend doing more wandering around town, getting some laundry done, and catching up on work and reading. I did get in a good run through the botanical gardens, around the opera house, and then up and over the big bridge and back.
On Monday, I left for a quick trip to Melbourne. After a few hours there, I was already bummed that I wasn’t spending more time there. I have to admit, that I’m a bit of a foodie, and good food is everywhere in Melbourne. It’s also a beautiful city with a great mix of new and old architecture.
I went for a nice run along the river Tuesday morning, walked around the city some more, and then met with a Melbourne area testers for a few beers and dinner in the evening (as well as some fun conversations).
After breakfast – and a bit more exploring – I headed back to Sydney in the afternoon. My return trip to Sydney had a few obstacles. The first was that for some reason, my domestic flight to Sydney left from an international terminal. This meant I had to go through some extremely long security and passport control lines. I was particularly ‘lucky’ on this trip and was pulled out of line twice for extra ‘checking’. And then it got interesting. For reasons too long to share here, I’ve always gone by ‘Alan’. My credit cards say ‘Alan’, my Microsoft id says ‘Alan’, my email address is alanpa…I bet most people don’t even know that my first name is ‘Donald’. Because I book my flights with my credit card, my boarding passes often (or nearly always) say ‘Alan Page’. I make sure signatures on my license and passport include my middle name, and this has never, in dozens of international trips, ever been a problem – until this flight. The agent questioned me for ten minutes, trying to get me to prove who I was. I think if I was actually travelling to another country, instead of a few hundred miles north, she wouldn’t have let me leave. But eventually, I made it onto the plane and back to Sydney.
My final trip to Sydney was uneventful. I spent most of the afternoon reading in a café, and then had dinner, and got some sleep before heading home.
The flight home was long (SYD->AUK->SFO->SEA), but uneventful. I’m home now, and although I managed to catch a bit of a cold, it’s good to be here and get back to my regular routine. My arrival at home was also nearly perfectly timed with a small fire drill at work, and that is accelerating my return to reality as well.
Overall, I had a fantastic time. I expect to return someday (whether it’s for a conference or for a vacation), and hope to meet many more great testers either way.
The wikipedia entry on test automation troubles me. It says:
Commonly, test automation involves automating a manual process already in place that uses a formalized testing process.
I can’t decide whether it bothers me because it’s wrong, or because so many people believe the statement is true. In fact, I know the approach in some organizations is to “design” a huge list of manual tests based on written and non-written requirements, and then either try to automate those tests, or pass the tests off to another team (or company) to automate.
This is an awful approach to test design. It’s not holistic. It’s short-sighted. It’s immature, and it’s unprofessional. It’s flat-out crummy testing. I frequently say, “You should automate 100% of the tests that should be automated”. Let me put it another way to be clear:
Some tests can only be run via some sort of test automation.
Some tests can only be done via human interaction.
That part (should be) obvious. Here’s the part I don’t think many people get:
You can’t effectively think about automated testing separately from human testing.
Test Design answers the question, “How are we going to test this?” The answer to that question will help you decide where automation can help (and where it won’t).
Here’s a screen shot of part of the registration form for outlook.com (disclaimer – I have no idea how this was tested).
Let’s look at two different ways of answering the “How will we test this?” question.
The “automator” may look at this and think the following.
The “human” may look at the same form and think this:
I’ll fire off a disclaimer now, because I’ve probably pissed off both “automators”, and “humans” with the generalizations above. I know there’s overlap. My argument is that there should be more.
In my contrived examples above, the “automator” is answering the question, “What can I automate?”, and the “human is answering the question, “What can I explore or discover?”. Neither is directly answering the question, “how are we going to test this?”.
I could just merge the lists, but that’s not exactly right. Let’s throw away humans and coders for a minute and see if we can use a mind map to get the whole picture together. Here’s what I came up with giving myself a limit of 10 minutes. There’s likely something missing, but it’s a starting point.
Now (and at no time before now), I can begin to think about where automation may help me. My goal isn’t to automate everything, it’s to automate what makes the most sense to automate. Things like submitting the form from multiple machines at once, or cycling through tables of data make sense to automate early. Other items, like checking for non-existent days or checking max length of a name are nice exploratory tasks. And then, there are ideas like foreign characters in the name, or trying RTL languages that may start as an exploratory test, but may lead to ideas for automation.
The point is, and this is worth repeating so often, is that thinking of test automation and “human” testing as separate activities is one of the biggest sins I see in software testing today. It’s not only ineffective, but I think this sort of thinking is a detriment to the advancement of software testing.
In my world, there are no such things as automated testing, exploratory testing, manual testing, etc.
There is only testing.
More musings and rants to follow.
I’ve been gathering some information on test automation systems recently and will probably have a flurry of posts upcoming on automation and related subjects.
Some observations as I browse what Bing has to tell me about the subject of Test Automation:
There’s more in my search results, but those three results alone say a lot about what’s wrong with the way (most?) teams approach test automation.
I expect I’ll write more eventually on all of these points, but my anecdotal experience, along with a few dozen web pages and articles tells me that there’s a lot of confusion in regards to test automation.
You see, test design (including the design of how test automation will execute) is really, really hard. It’s hard to write sustainable and reliable tests, and it’s hard to write trustworthy tests. Double the effort required if those tests involve UI automation. Designing good tests is one of the hardest tasks in software development. That’s worth saying again. Designing good tests is one of the hardest tasks in software development.
Compared to everything else in the equation, the “writing code” part of test automation is easy. Massively easy. Almost brain-dead easy. When I compare writing code to composing music, I talk about the creative and challenging aspect of melody, creating a texture and mood, and figuring out where notes and space between notes help with both. There’s also the rote part of the job – chord voicings, doubling, and other parts of filling in the score. The melody and texture choices are the test design of music composition – filling in the rest of the parts is the coding part. Sure, there’s creativity in a countermelody, as much as there’s creativity in a cool algorithm, but it’s still the rote part of the activity.
I fear that testers are worrying too much about the effort and skills required to automate tests – and not enough about what it takes to design reliable, portable, accurate, and trustworthy tests that actually matter.
Tell me I’m wrong.