{"id":840,"date":"2014-04-10T09:33:13","date_gmt":"2014-04-10T16:33:13","guid":{"rendered":"http:\/\/angryweasel.com\/blog\/?p=840"},"modified":"2014-04-14T10:06:14","modified_gmt":"2014-04-14T17:06:14","slug":"users-usage-usability-and-data","status":"publish","type":"post","link":"https:\/\/angryweasel.com\/blog\/users-usage-usability-and-data\/","title":{"rendered":"Users, Usage, Usability, and Data"},"content":{"rendered":"

The day job (and a new podcast) have been getting the bulk of my time lately, but I\u2019m way overdue to talk about data and quadrants.<\/p>\n

If you need a bit of context or refresher on my stance, this post<\/a> talks about my take on Brian Marick\u2019s quadrants (used famously by Gregory and Crispin in their wonderful Agile Testing book); and I assert that the the left side of the quadrant is well suited for programmer ownership, and that the right side is suited for quality \/ test team ownership. I also assert that the right side knowledge can be obtained through data, and that one could <\/em>gather what they need in production \u2013 from actual customer usage.<\/p>\n

And that\u2019s where I\u2019ll try to pick up.<\/p>\n

\"image\"<\/a><\/p>\n

Agile Testing<\/a> labels Q3 as \u201cManual\u201d, and Q4 as \u201cTools\u201d. This is (or can be) generally<\/em> true, but I claim that it doesn\u2019t have<\/em> to be true. Yes, there are some synthetic tests you want to run locally to ensure performance, reliability, and other Q4 activities, but you can get more actionable data by examining data from customer usage. Your top-notch performance suite doesn\u2019t matter if your biggest slowdown occurs on a combination of graphics card and bus speed that you don\u2019t have in your test lab. Customers use software in ways we can\u2019t imagine \u2013 and on a variety of configurations that are practically impossible to duplicate in labs. Similarly, stress suites are great \u2013 but knowing what crashes your customers are seeing, as well as the error paths they are hitting is far more valuable. Most other \u201cilities\u201d can be detected from customer usage as well.<\/p>\n

Evaluating Q3 from data is \u2026interesting. The list of items in the graphic above is from Agile Testing, but note that Q3 is the quadrant labeled (using my labels) Customer Facing \/ Quality Product. You do Alpha and Beta testing in order to get customer feedback (which *is* data, of course), but beyond there, I need to make a bit larger leaps.<\/p>\n

To avoid any immediate arguments, I\u2019m not<\/span><\/em> saying that exploratory testing can be replaced with data, or that exploratory testing is no longer needed. What I will<\/em>\u00a0 say is that not even your very best exploratory tester can represent how a customer uses a product better than the actual customer<\/em><\/strong>.<\/p>\n

So let\u2019s move on to scenarios and, to some extent, usability testing. Let\u2019s say that one of the features \/ scenarios of your product is \u201cUsers can use our client app to create a blog post, and post it to their blog<\/strong>\u201d. The \u201ctraditional\u201d way to validate this scenario is to either make a bunch of test cases (either written down in advance(yuck) or discovered through exploration) that create blog entries with different formatting and options, and then make sure it can post to whatever blog services are supported. We would also dissect the crap out of the scenario and ask a lot of questions about every word until all ambiguity is removed. There\u2019s nothing inherently wrong with this approach, but I think we can do better.<\/p>\n

Instead of the above, tweak your \u201ctesting\u201d approach. Instead of asking, \u201cDoes this work?\u201d, or \u201cWhat would happen if\u2026?\u201d, ask \u201cHow will I know if the scenario was completed successfully?<\/strong>\u201d For example, if you knew:<\/p>\n