Don Rickert Design

Usability Evaluation

Usability Evaluation can simulate relatively unobtrusive observation of semi-realistic behaviors. Generally, Usability tests do a good job of uncovering egregious problems with conventional quality--things such as bad links, buttons that stick, etc.

The 5 Levels of Usability Assessment

Following the practice of Dr. Robert Bailey of Computer Psychology, Inc., we use up to 5 levels of Usability Assessment, depending on the project. These levels of assessment range from a simple Usability Review to a full-blown scientifically controlled comparison of two products or two versions of a single product.

Level 1 Inspection : review by two experts against common usability rules of thumb (i.e. 'heuristics')

Usability Inspection is used to make a quick assessment of the quantity and magnitude of commonly-observed usability problems. Because actual users or consumers are not part of Usability Review, it must never be the sole criterion on which the goodness of a product is based.

It is critical to keep in mind at all times, that real users always turn up surprises that are missed by the Reviewers.

Level 2 Expert Review: review by at least 4 experts against a comprehensive list of usability guidelines

Expert Review is used to make a quick assessment of the quantity and magnitude of commonly-observed usability problems. Because actual users or consumers are not part of Expert Review, it must never be the sole criterion on which the goodness of a product is based.

It is critical to keep in mind at all times, that real users always turn up surprises that are missed by the Experts.

Level 3 Expert Review: review by 12 -16 (combination of usability experts and domain experts). The larger number of reviewers allows statistically valid assertions (but still preliminary) to be made about the product.

Level 4 Usability Test: formal test involving 12–16 actual or potential users

Level 5 Usability Test: A scientific comparison of two versions or two different products

Usability Evaluation Planning

The idea is to ensure that usability test protocols are based on insights discovered through real world observation and depth interviews. This way, the usability tests will test the right tasks

Lacking a clear notion of who the users are and what they do, how could one even know what questions to ask or what to test in the usability lab? In science jargon, asking irrelevant or just plain wrong research questions is called 'Type III Error'.

The common practice of conducting usability testing prior to rigorous Ethnographic study, such as Field Observation and Depth Interviews, is a principal reason for so much usability testing being a waste of time and money.

By analogy, conducting a usability test before Personas and Scenarios are developed is like booking a recording studio before any of the songs are written--very expensive and very little valuable output.

This is why it is essential that Usability Evaluation Planning be done as an integral part of Researching the Opportunity and take full advantage of the insights from Ethnography, Initial Market Research, Human Factors and Persona/Scenario Refinement.

Why Surveys and Usability Testing are Often a Waste of Time and Money

A recent rant in Interactions Magazine (January - February 2006) asks the question: are you doing 'User Research' or 'Loser Research'?  The author is not referring to researching people who are 'losers' but rather researchers who are not doing good work-- 'Loser Researchers'. That article inspired our post below.

Attempting to develop any kind of product without a clear notion of who the users are and what they might do with the product is a very bad idea. Furthermore, even starting other types of consumer research, such as surveys and usability testing before the product opportunity is fully understood, or before personas (i.e. users) and scenarios (i.e. the things they do) are defined is ineffective. Lacking a clear notion of who the users are and what they do, how could one even know what questions to ask or what to test in the usability lab? In science jargon, asking irrelevant or just plain wrong research questions is called 'Type III Error'.

The common practice of conducting usability testing prior to rigorous Discovery Research, such as Field Observation and Depth Interviews, is a principal reason for so much usability testing being a waste of time and money, and this hurts the credibility of Usability Engineering as well as its related professions.

By analogy, conducting a usability test before scenarios are developed is like booking a recording studio before any of the songs are written--very expensive and very little valuable output.