08 Nov 2002; by Dennis G. Jerz
This document is intended to help beginners design questions to help them conduct a good usability testing session.

Usability Testing is Not Opinion Gathering.

If you plan to show your document to your subjects and ask them questions like, "Is my document well-organized?" and "What do you think of my table of contents?" then you aren't going to get any useful information.

	I've worked hard on this document; can you help me meet my deadline? Please tell me what you like about it -- I'll feel so affirmed and flattered. If you get stuck, or hesitate, I will tell you what to do, so we can get through this testing faster. If you start making any serious suggestions, my face will cloud over with a look of pain, since I don't want to hear anything but praise.
Remember on The Andy Griffith Show, when Aunt Bea gave Andy a sample of her new apple pie recipe, and it was awful but he couldn't bear to tell her, so she ended up being completely humiliated by the judges at the county fair? (Well, maybe it was pickles and maybe it was Alice on The Brady Bunch, but you get the idea.) Most volunteer testers are nice people who won't want to say anything negative about your work. But you want them to find flaws in your prototype, so that you can fix them before the deadline.
	I'm trying to determine the strengths and weaknesses of this document. I will read you some questions and keep track of how long it takes you to answer each one. Just do whatever comes naturally. If you should run into problems, don't feel bad -- I'm testing the document, not you. Feel free to talk out loud and tell me whatever comes to mind. If you have any suggestions, I'll make sure the author of the document sees them. (After the test is over.) I really learned a lot from watching you -- thank you!
You'll get better results if the testers don't know you are the person who designed the document they are testing. With your eyes open and your mouth shut, you need to watch what your testers do as they confront your document. Since you've left plenty of time in the production cycle (right?), you are ready and eager to solicit their input and act on what they say.

In a good usability test, your testers will use your document to do whatever your real users want to do. Rather than simply ask your testers to "look at" your document and tell you what they think, come up with a short list of definite tasks -- finding a bit of information, collecting information from different pages, making judgments about the content, etc.

1. Keep the Test Short

If the test is long, few subjects will bother with it. I once had a student who mailed out a thousand questionnaires, and got zero responses back.

Try out a longer test on a small number of subjects, and then cut those sections that don't result in useful data. Once you've refined your usability test, you can try it out for real. (I encourage my students to ask me to look at their usability tests before they invest time in using them. It's no fun to find out, at a late stage in your project, that you've gathered no useful information.)

2. Plan to Quantify Your Results.

When gathering data, it's easy to ask questions like "Did you think the navigation was clear?" You might tabulate the "yes" and "no" answers, but how will you quantify responses to a general question like "What did you think about the table of contents?"

When you do ask general opinion questions, use a seven-point Likert scale, in which you make a neutral statement and ask for testers to rate their responses. For example:

This document is clear.

1. Disagree Strongly
2. Disagree
3. Disagree Somewhat
4. No Opinion
5. Agree Somewhat
6. Agree
7. Agree Strongly

You now have some numerical data to work with, allowing you to identify trends such as the following:

In the first usability test, subjects reported an average score of 5.2 for question 1, indicating only a very slight agreement; this agreement strengthened to 5.7 in the second usability test, and 6.1 in the third.

Your readers probably won't want to hunt through paragraphs like the one above; use a table to present your results efficiently. As you make changes to your document and re-test it, you can quantify the change in the usability factor.

Table 1: Responses to Subjective Statements

	Test 1	Test 2	Improvement
1: "This document is thorough."	4.2	5.0	+19%
2: "This document is easy to understand."	3.0	4.5	+50%
3: "The site was too complex."	4.0	3.0	+25%
Average	3.73	4.17	+31%

If you also tabulated your results for "Task Time", "Task Errors" and "Memory", you could then calculate the average improvement for each section.

(Note that, for "Task Time" and "Task Errors", a lower number constitutes a better score. My example below suggests one way you can make that concept clear to your reader.)

Table 2: Average Usability Change

	Test 1	Test 2	Improvement
Subjective	3.73	4.17	+31%
Task Time (sec)	120	95	+26% *
Task Errors	3.0	1.5	+50% *
Memory	3.0	4.0	+33%
Avg. Usability Change			+35%
* Note: For these criteria, a lower score on the second usability test corresponds to an increase in usability; the percentage value in the "Improvement" column has been inverted to reflect this relationship.

3. Ask Subjects to Prioritize, Rank, or at Least List.

Instead of asking questions that your subjects can answer "Yes" or "No", ask your subjects to prioritize, rank, or at least list.

For example:

• What are the three things you noticed on the home page?
• Now that you have read the brochure, what are three important facts that your remember about the content?

If many of your test subjects mention some part of your document that you hadn't thought was very important, that may be a sign that your testers could teach you a lot about that section.

4. Sequence Your Subjective Questions from General to Specific.

A subjective question is one that asks for the tester's personal opinion:

• How likely are you to trust the webmaster?
• Which section of the website needs the most attention?

Finding out what users think they know can help designers predict trouble spots. For instance, if 80% of test subjects rated a set of instructions "easy to follow," but only 40% were actually able to complete the task correctly, then the instructions are seriously flawed because they are giving testers a false sense of confidence.

Begin with general questions that ask your testers to come up with details; then, ask the specific questions in which you tell testers what details to think about.

By simply mentioning something in a question, you call attention to it. It's also possible to call attention to something indirectly -- if I first asked you to think of a monkey, and then asked you to think of a fruit, you'd probably be more likely to think of a banana, and the resulting list of "most popular fruits" would be skewed.

Likewise, if you ask specific questions about parts of your document, all the answers that follow will be skewed.

	Did you think the navigation on the website was good?
	Calling attention to the navigation with a question that is fishing for a "yes" answer will not give you an accurate view of the usefulness of your document.
	What did you think of the navigation on the website?
	The above example is at least honest, because it doesn't encourage the reader to praise the navigation, it still artificially calls attention to it.
	List three good things about the website and three bad things.
	Maybe the navigation will turn up in one or the other list; maybe the testers will focus their attention on different things entirely.

The solution is to ask general questions first, and then ask specific questions.

5. Move Beyond Opinion

If you ask yes/no questions like "Did you like X?" then you are only measuring an opinion. As long as you also record things like how long it takes users to perform certain typical tasks, how accurate their answers are, and how much they remember a short time after they've used your document, then it's perfectly fine to gather subjective information. For instance, if a user answers "Yes" to "Do you understand this document," but misses too many memory recall questions, then we can hypothesize that the document gives the user a false sense of confidence.

There's another problem with yes/no opinion questions like "Did you like the document?". The answers aren't terribly useful. Even if my tax forms are designed really well, I'll never like them.

You might be thinking, "Oh, so what I need to do is ask more specific questions like 'Is this document well organized?'" But the average person probably isn't trained to evaluate a document's design. User testing asks you to observe what people do, and measure their performance. You are the expert who will use the resulting data to make decisions about design and content, and user opinion is only one piece of the puzzle that you'll have to assemble.

6. Ask Questions in Pairs

Asking users a question like "Could this part of the document be improved?" doesn't get right at the core of the problem. Maybe that part of the document is great, but the subject can think of a brilliant plan to improve usability by .oo5%. In that case, the subject would answer "Yes, this passage could be improved," but that would still not give you any useful information.

It's often a good idea to ask users to make a snap judgment, and then ask them to explain their answer. Asking for the judgment without explanation will close off an avenue of communication; asking for a short essay won't give you information you can easily quantify.

1) Respond to the following: In my free time, I am [blank] to read a book. very likely likely somewhat likely neither likely nor unlikely somewhat unlikely unlikely very unlikely 2) Why did you answer the way you did?

If your multiple-choice and short-answer questions work together, you get both

• a number (in this case, a rating on a scale of 1-7) you can plug into a chart and
• words (the response to a short-answer question that you can quote in your report)

7. Plan a Series of Tests (with Different Testers)

Your first usability test measures the "before". Use the results to guide your efforts to improve your document, and then test again. (And again.)

If for, example, testers took on average 2 minutes and 24 seconds to do all five tasks, they got four out of five tasks right, they answered 2 out of 4 memory questions right, and their subjective reaction was 5.2, then those answers become the "control" group. There's nothing magic about these numbers -- they're just an arbitrary starting point.

Find different volunteers for each set of tests.

If your goal is to determine how a first-time user reacts to whatever you are testing, then re-using the same volunteers will not provide you with very much useful information. Yes, the volunteers will be able to tell you whether they do or don't like the changes you implemented, but volunteers who have slogged their way through your prototype are no longer representative of the general public -- they have been contaminated (so to speak) by their knowledge of your developing project.

8. Plan Realistic Tasks

Your usability testing should emulate, as closely as possible, the environment in which your end users will interact with your document.

A simple scavenger hunt is only marginally useful. If you give your testers a list of things to find ("the 'Comments' page" and "the e-mail address for the help desk," for example), and then time how fast they can find it, that's only a marginally useful task. (You could too easily improve their time on that task by making the specified information stand out more on the page, but that might make it harder to find the rest of the information on your site.)

When an oil rig starts tipping over in a storm, the oil riggers don't have someone hovering over their shoulder telling them what chapter they should turn to. Instead, the oil riggers will start with what they know -- they have a problem; if they have reason to suspect that your document holds the answer to their problem, they'll be motivated to read (and use) your document.

So, instead of announcing the solution to the problem and timing how long it takes them to find the page that contains the solution, you might state a problem, and watch how people use your document in order to solve it.

Dennis G. Jerz
08 Nov 2002 -- first posted
17 Nov 2002 -- minor revisions
04 Dec 2002 -- minor revisions
10 Mar 2008 -- minor corrections

---

Case Study: A Pamphlet

If I were testing a pamphlet that described the major programs at my school, I would think of a handful of ways that I would expect a user to use that pamphlet. Where would a person pick it up? How likely would that person keep it for future reference? How frequently will the pamphlet be updated? I would first find about two volunteers (who represent the target audience as closely as possible -- if the target audience is incoming freshmen, then the volunteers shouldn't be graduating seniors), and watch them interact with the document I'm about to test. I'd then talk with them about their initial efforts to use the document to accomplish real tasks. Based on what I learned, I would come up with maybe five specific tasks, ranging from very simple (what's the phone number you call to get more information?) to complex (what's the major that requires the largest number of non-English courses? [the answer to a question isn't in one place, it requires the reader to flip back and forth and compare]); I would then find five subjects for the first "real" usability test. I would how long it took them either to find an answer or to give up. (I'd ask them to stop if they haven't found it in a pre-determined amount of time). I'd then take the brochure back, and ask them to tell me what three things they remember about it. I'd then ask them what they thought of the brochure in general (having them "agree" or "disagree" on a seven-point scale to questions such as "This brochure was helpful," or "This brochure was complete."), and invite them to make any suggestions. After they've made their own general suggestions, I would ask specific questions about parts of the document I'm really interested in examining. I would then consider all the ways I could possibly improve the document, and rank them according to how much effort it would take to implement them, and how important they are to the project. Obviously, I'd do the "easy and also important" fixes right away, and then consult with my client and/or supervisor about the rest. --DGJ