Don Rickert Design

Why Surveys and Usability Testing are Often a Waste of Time and Money

A recent rant in Interactions Magazine (January - February 2006) asks the question: are you doing 'User Research' or 'Loser Research'?  The author is not referring to researching people who are 'losers' but rather researchers who are not doing good work-- 'Loser Researchers'. That article inspired our post below.

Attempting to develop any kind of product without a clear notion of who the users are and what they might do with the product is a very bad idea. Furthermore, even starting other types of consumer research, such as surveys and usability testing before the product opportunity is fully understood, or before personas (i.e. users) and scenarios (i.e. the things they do) are defined is ineffective. Lacking a clear notion of who the users are and what they do, how could one even know what questions to ask or what to test in the usability lab? In science jargon, asking irrelevant or just plain wrong research questions is called 'Type III Error'.

The common practice of conducting usability testing prior to rigorous Discovery Research, such as Field Observation and Depth Interviews, is a principal reason for so much usability testing being a waste of time and money, and this hurts the credibility of Usability Engineering as well as its related professions.

By analogy, conducting a usability test before scenarios are developed is like booking a recording studio before any of the songs are written--very expensive and very little valuable output.

New Product Development Needs More than Just Ethnography

After reading some of our posts, one could get the impression that we believe Ethnography is the magic bullet for increasing new product success to something more than the typical 3% to 5% probability. This is not quite the case. Believing that Ethnography is the single answer to product success would be just as ridiculous as the apparent belief held by firms that push their brand of usability testing, or focus groups, or surveys, or design, etc. to the exclusion of all other approaches.

What we do say is that, unless you deal with the so-called 'fuzzy front end' of New Product Development (the part that comes before Implementation--whatever you want to call it) in a disciplined manner, you should expect about a 3% to 5% probability of success.

There are four primary core competencies required in order for the front end of NPD to yield actionable information: Research, Marketing, Design and Engineering. Ethnography falls under the Research category. The four core must competencies work collaboratively in a process that can be summarized in the following diagram (click to enlarge it):

There are a number of influences, not the least of which is our own experience, on which we base our approach to NPD. Click the 'continuation' link below to read about these.

Interesting Description of What an Ethnographer is

In a recent Business Week article, The Science of Desire, Ethnographers are described as:

'...a species of anthropologist who can, among other things, identify what's missing in people's lives -- the perfect cell phone, home appliance, or piece of furniture -- and work with designers and engineers to help dream up products and services to fill those needs.'

The rather lengthy article is full of informtion of how companies that 'get it' are using Ethnographers and other Social Scientists to observe the real world in search of inspiration for new products that people actually need and want. I can't resist including the closing sentence in this posting:

'...with anthropologists in ascendance, engineers -- and everyone else -- had better get used to it.'

If it makes you feel any better, in addition to being Ethnographers, Brad Wiederholt is an Engineer (actually, a Computer Scientist) and I (Don Rickert) am a Market Researcher & Industrial Designer. We look forward to comments from engineers and everone else :-)

DR

So You Think Ethnography is Just a Fad?

There has been a lot of press on ethnography as the new core competency for new product development of late. Business week has a slew of recent articles and online slide shows. Ethnography firms looking for a way to state their value proposition to 'the suits' need look no further! (need to finish this posting quickly so we can start re-crafting our own 'elevator speech.')

For a concise slide show of some recent contributions of ethnography to product breakthroughs by identifying actual consumer needs (by Spencer E. Ante), click on the link at the end of this posting. See ethnography-driven new product introductions from:

Motorola (Product: new cell phone)
Sirius Satellite Radio (Product: Portable satellite radio)

Intel (Product: PCs for emerging markets)

Citigroup (Product: New payment service and a device called PayPass)

OXO (Product: Professional grade hardware tools--from pizza slicers and bottle openers to hammers!) Note: I saw these new OXO tools just yesterday while buying Dremel bits at Lowe's for a new WARP3 Insights product model.

Business Week Slide Show on Ethnography Hits

The “Closed-to-Outsiders” Sub-Cultures Dr. Don Rickert Has Gained Access To

All of the following groups, in varying degrees, are essentially closed to “outsiders”. Dr. Don Rickert has the innate skill, which he has cultivated in his professional life, to gain trust and access to the following and other groups in order to learn more about their unmet needs, their desires and what is appealing to them at an emotional level. Don has done everything from sanctioned drag racing, to learning to row (and joining a team) to spending days on end in neurosurgery operating rooms.

Clients that commissioned Ethnography studies involving the groups listed include:

• Telecommunications & Information Technology company
• NIST (National Institute of Standards & Technology)
• Sporting equipment manufacturer
• Musical Instrument company
• New Product Development (NPD) Firm
• US Air Force
• A major  US  Grocery  chain
• Electric Power Research Institute
• Enterprise Software company
• CRM Software company
• Enterprise Software company
• Telecommunications company
• Several Large Financial Services Providers

 

Medical

• Medical Students
• Medical School Faculty
• Neurosurgeons
• Radiologists
• Hospital Information Technology Managers and Staff

Music

• Youth Prodigy Musicians (likely to earn scholarships to top music schools such as Berklee, Julliard and Peabody) and their parents
• “A-List” Professional Musicians (i.e. recording artists)
• Top “Old Time” and Bluegrass Musicians (Fiddlers in particular)
• Contra and Square Dancers
• Cajun Music and Traditional Dance Enthusiasts

Graphic Arts

• The “Folk” and “Outsider” Art culture (gallery owners as well as Artists)

Active & Extreme Sports

• Competitive Rowers
• Rock Climbers
• Martial Arts (Mixed Martial Arts, Judo, Tai Kick-Boxing, etc.)
• High-performance large boats
• Sailing Enthusiasts

Historic Re-enactment Hobbyists

• Civil War Re-enactors
• Scottish Highlander Military Re-enactors
• Colonial Period Re-enactors

Automotive Culture

• Amateur Drag Racing

Financial Services (Banks)

• Bank Senior Managers
• Bank Tellers
• Bankers
• Bank Call Center Staff Members
• Customer Service Representatives

Telecommunications/Cable TV

• Cable TV/Broadband Installers

Miscellaneous

• Day Care Center Managers and Staff
• Cleaning Business Owners
• Niche Craft Business Owners

Downloads from Ethnography in the Fuzzy Front End of Product Development

For those who attended Jay Heisler's (of UPS) and Don Rickert's (of Wiederholt & Rickert Partners, LLC) presentation on Ethnography in the Fuzzy Front-End of Product Development, thank you--you were a wonderful audience. We have broken the PowerPoint into four parts for download, as the presentation is quite large.

If you have any questions or problems with downloading, please do not hesitate to call Don Rickert at 404-713-3750 or email him at [email protected].

• Download Main Presentation (PowerPoint)
• Download Examples of Traditional Techniques images (linked PowerPoint)
• Download Observations in a Mega-Store images (linked PowerPoint)

• Download What Does Ethnography Look Like? images (linked PowerPoint)

Acrobat Version of Ethnography in the Fuzzy Front End of Product Development

A number of attendees at the recent American Marketing Association presentation by Jay Heisler (UPS) and Don Rickert (Wiederholt & Rickert Partners) and colleagues have requested an easily printable version of the presentation. Here it is (a zipped Acrobat file):

Download Ethnography in the Fuzzy Front-end of New Product Development print version 5-22-06 (zipped)