The Science of Usability Testing

The Science of Usability Testing

Jean E. Fox. 2015. (View Paper → )

Usability testing has evolved from the stringent methods of experimental psychology, to less controlled, more qualitative tests, to the wide variety of methods used today. As the methods have evolved, researchers have studied many aspects of usability testing with the goal of better understanding how to best implement, plan, conduct, and interpret tests.

As well as the insights below, the paper gives you a feel for how usability testing has evolved over time.

1. Number of participants
• There are diminishing returns to increasing participants. Around four to five participants can reveal about 80% of usability problems, with diminishing returns for additional participants.
• You’ll need more participants if you’re testing a complex system, or if your user group is not heterogenous.
• General guidelines suggest 5-10 participants per user group for qualitative tests and 20-30 for quantitative tests.
2. Number of Trained Observers
• A single observer might miss issues that multiple evaluators could identify.
• The number of trained observers (evaluators) significantly affects the identification of problems in usability testing. The lesson here is to involve the team.
• Adding more evaluators can be as effective as adding more participants, especially when participant recruitment is challenging or time is limited.
3. Use of the Think-Aloud Method

• The think-aloud method can influence test results; for example, it can make participants more aware of their thought processes, potentially leading to faster problem-solving.
• Different approaches to think-aloud (e.g., traditional vs. relaxed methods) can have varying effects on aspects like task time and mental workload.
• While think-aloud is a powerful tool for identifying usability problems, its implementation should be carefully considered, especially in tests focusing on quantitative measures.