Usability Expert Stresses Qualitative Approach
By Maria Trombly, The SDP Documentor, Dec. 1999

It doesn't have to cost a lot to do quantitative usability analysis.

According to Chauncey Wilson, the software development manager at BMC Software, it can take as little as your eyes, a pencil, and a watch. Wilson also teaches a course at Bentley College in user-centered design and before his promotion six months ago, he worked on the usability side at BMC.

"I'm a believer in both the quantitative and the qualitative," he said. "But we need to measure usability. This is a place where I have a bit of fervor."

Wilson said he likes to start a usability program with a baseline test, and tries to improve the usability rating by 30 or 40 percent with each generation of the product.

Benchmarking is one of Wilson's favorite topics, he said. "If you go out to Microsoft and their usability labs, they do a lot of this," he said. "One company that I worked for had a product that supposedly took eight hours to install. In fact, it took three days and we had to buy cots for people."

During his presentation to the SDP on October 25, Wilson outlined a number of usability tests, both quantitative and qualitative, that he uses in his lab or teaches in his course.

A summary test is one in which the user reads a page of documentation, then is asked to write down the critical concept on a piece of paper.

There are unlimited variations to this test, Wilson said.

A user interface race is one in which experts with secure egos race each other to accomplish a certain task. "I see a lot of design that's usable but not efficient," Wilson said. "This gets at that a little bit."

Coaching is a useful technique in situations where the users aren't willing to talk out loud or give honest opinions of a product.

Incredibly intelligent help is a technique similar to coaching, in which the user is given little bits of information in order to determine which bit is the critical bit.

A critical incident test is one in which the user notes both the successes and failures of a product. This test can be performed in a lab, by telephone, or even by questionnaire.

A read-and-locate test is one in which users are asked to find answers to cmestions in a help file or document. A good source for questions for the read-and-locate test is the telephone technical support database.

Help desk mining is a good tool in many situations. Wilson said. "I usually find that between 30 and 60 percent of the problems people call about are usability related," he added.

Another source for usability problems are company training classes. where students can find problems ranging from typos to missing tasks. "I've always liked that method," Wilson said. "But it recquires some tact and negotiation with training groups."

Not all usability testing recquires the use of guinea pigs drawn from the pool of potential users. Some tests can be performed by staff, though such tests can miss many of the problems that real users can pick up. An evaluation against a set of general principles, for example, can be performed with in-house experts who evaluate the product according to a list of guidelines. These kinds of tests can pick up between 20 and 80 percent of the problems in a product.