Eight is Not Enough

“Today’s web sites, particularly e-commerce sites, can be more complex than standard software products that often confine users to a very limited set of activities. Web tasks are also vastly more complex than those users have with most software applications. For example, our tests asked users to complete shopping tasks. No two users looked for the same product and no two users approached the site in the same way. The tasks were dependent on individual user characteristics and interests. Because of the increased complexity of web sites, it’s understandable that more users are needed to detect the majority of usability problems.” Christine Perfetti and Lori Landesman

—Eight is Not EnoughUser Interface Engineering)

Perfetti & Landesman challenge the budget usabilty testing rule of thumb — that about 8 users will catch most of your usabilty testing problems. My intuition tells me that if you have the budget to hire 10 users, you’re probably better off stopping the test after 5, applying the results, and then asking the next 5 users to test your revision. Perfetti and Landesman point out that completing an online purchase is a very complex action, with no two users having the same preferences or search patterns. But because they were only finding 35% of the usability errors with their first 5 users, I’d suggest that they should perhaps have made their usability tests more complex — thus getting more results from each tester. Nevertheless, the authors conclude with a sensible argument for what Tognazzini called “Close Coupled Testing“. (I miss the AskTog columns.)