|
|
The Quick, the Cheap, and the Insightful
Presented by Dana Chisnell and reviewed by Richard Mateosian
Dana Chisnell wrote the book on usability testing, but she knows thatinsisting on going by the book can result in doing nothing at all. Usability testing in the wild is her term for small scale ad hoc testing that applies the principles but skimps on the rigorous details. It can give you useful qualitative results at low cost.
Dana is a well known usability expert and an STC Fellow. She blogs on usability testing at UsabilityTestingHowTo.blogspot.com. She has recently collaborated with Jeffrey Rubin to bring out a second edition of Handbook of Usability Testing, which you can buy at almost 40% off (as of December 28) from Amazon.com.
Usability testing, like most testing, is a sampling process designed to gather information about how well something you are interested in performs. Handbook of Usability Testing lays out a sequence of steps for "classic, with everything" usability testing:
- Develop a test plan.
- Choose a testing environment.
- Find and select participants.
- Prepare test materials.
- Conduct the sessions.
- Debrief with participants and observers.
- Analyze data and observations.
- Create findings and recommendations.
Each of these steps merits its own chapter in the book, so this is a pretty heavy-duty process. In most situations, faced with a choice between following this process and doing nothing, you're likely to do nothing. Drawing on her extensive experience, Dana shows how you can throw away the book (but buy it first, of course). She identifies parts of this process that can yield value at low cost. Crossing out lines and words in the above list and adding a final clause, she arrives at the following:
- Develop a test plan.
- Find participants.
- Conduct the sessions.
- Debrief with observers and decide together.
This is usability testing in the wild. Conducting sessions boils down to a simple process. Sit next to someone (in their space, not in a lab). Watch them do stuff. Notice what happens -- hesitating, comments, questions, body language, behaviors. Don't teach -- ask open-ended questions like Why? How? What?
Dana points out that you can do usability testing in a variety of places and ways. She makes a distinction between subjects -- the designs you are testing -- and participants -- the people you are watching as they use the design. Prospective participants are friends, family, colleagues, or even hired temporary workers. They are people waiting in the company lobby, enjoying a drink at a cafe, attending a trade show, or sitting next to your middle seat on a long flight. They can always say no, but it rarely hurts to ask. You can listen in on customer service calls or review call logs. You can talk with sales reps or follow user forums and wikis. You can even view browser histories, bookmarks, and email trails.
Dana told us the story of the pro bono work that she and Nancy Frishberg did for the Marin County Registrar of Voters, Elaine Ginnold. They were invited in at the last minute. The work and the final report all had to happen in one day. Dana has been working on ballot usability for a while and has studied troubled elections, so she wasn't starting from scratch. Still, the time limit was short.
Here are Dana's rules for projects like this one:
- Say yes.
- Trust the team and participants.
- Don't let the team or stakeholders block.
- Work to the top of your collective intelligence.
In this case they used the ballots and instructions at hand, focused on one thing at a time, and had the participants generate and collect data. They debriefed everybody at the end. The objective was to find the most significant problems with the current ballot design. Things that might cause a recount or cost money in other ways are significant. So are problems that make it difficult to determine what voters intend by the way they mark their ballots. The team couldn't change the ballots, but they could change the instructions. The Registrar provided participants. The team drafted passers by to be observers. They focused on one issue at a time. It only takes a few participants to uncover a problem. The team made useful suggestions at the end of the day.
There is no such thing as a free lunch. Usability testing in the wild produces results inexpensively, but not without risk. Reliable results require a large, unbiased sample of participants. A small ad hoc sample may give useful qualitative results, but not good, quantitative data. The biggest tradeoff is that if you hold out for full-blown, rigorous usability testing, you might get nothing at all. So don't be afraid. Grab some participants and put them to work. The quality of your designs will benefit.