Everyone starts out as a beginner. Many of us learned how to run a usability test from books or simply on-the-job training. There are many resources to learn the basics, such as Steve Krug’s Rocket Surgery Made Easy and Joe Dumas and Beth Loring’s Moderating Usability Tests: Principles and Practices for Interacting. The hard skills are fairly easy to master, but those soft skills needed to conduct unbiased facilitation take lots of practice. It is even harder when the participant has difficulty focusing their attention and simultaneously giving you the feedback you’re seeking. Practicing the soft skills of persuasion will improve communication with participants and enable you to gain valuable insights even during difficult situations.
After decades of usability testing, I noticed patterns in the behaviors of participants that made facilitation challenging. At Bentley University, we gave them personality names so we could quickly identify them and be ready with a response to keep the information flowing.
I’ve been fortunate to have mentored more than 75 inexperienced or out-of-practice researchers, from grad students to corporate employees, in the past 15 years. The best way to train people to use their soft skills to manage challenging participants is to practice facilitation in a controlled environment. I call it the Three Stooges Mock Usability Test.
Here’s how it works:
- The new facilitator creates a usability test script to evaluate a website
- You (or another qualified researcher) act as the participant, starting out as a “cooperative participant”
- Record the session to aid in the debrief of their performance
- Once the facilitator is comfortable you fall into one of several challenging but common participant personalities
- Use the checklist provided below to guide your bad behavior and track when the facilitator misses the cue
- After the session, review the checklist and advise on catch phrases to use
- Where appropriate, go back to the recording to show them exactly where they missed the cue
As researchers, we wish all participants were easy to facilitate. The perfect participant would be articulate, insightful, easy to talk to, and stay on task. Unfortunately, they’re not all this perfect.
Let me introduce you to some of our more popular difficult participant personalities; see if you recognize any of these!
The Talker
The Talker personality gives too much information and makes it hard for the moderator to ask their questions. They are also known to go off task from time to time, making it especially challenging to bring them back to the script. In the end, you get lots of great data from this personality, but you may feel exhausted when it is over!
The Test
- Excessively like or dislike something to try and get the facilitator to agree with you. They should remain unbiased and neutral.
- Talk fast and go way off track of where the task wants you to go. See how long it takes the facilitator to bring you back on task.
The Cold Fish
The Cold Fish is the complete opposite of The Talker. They don’t give away any information as they work, making it a challenge to get them to articulate their feedback. They give short answers that are sometimes very vague. This keeps a moderator on their toes, asking open-ended, probing questions.
The Test
- Be silent for a few minutes at several points in the session as you complete tasks. See how long it takes the facilitator to probe for your feedback.
- Dislike something without giving a reason. You could just grunt in disgust or explicitly state, “I don’t like that.” The facilitator should be asking you why you don’t like something.
- Leave them hanging with statements like, “That’s not what I would call it.” Again, we’re looking for follow-up questions to understand why.
The Self Blamer
The Self Blamer is the person who doubts they are being helpful in the session. They think all of their task failures are their fault, often stating, “It’s me. I’m being stupid,” or “Once I figured it out it was easy.” They are also the ones who, after struggling with a task, report how easy it was afterwards.
The Test
- Struggle to complete a task then blame yourself for poor performance. We want the facilitator to reassure you it is the user interface that is bad.
- Redirect your feedback toward a non-target user with a statement like, “My grandmother would have a hard time using this.” The facilitator should ask a follow-up about how well it works for you rather than the non-target.
Once the mock session is over, review your notes on how well your partner moderated. Point out in the video where they did well, asked leading questions, or missed opportunities for further probing. Facilitation practice helps them to be prepared to think on their feet. The more they practice tough testing situations, the more comfortable they will be when it really counts.
This simple checklist helps to run a mock session and act as these challenging participants along with advice of associated catch phrases:
| Participant Does This… | If the Facilitator does this… | Advise this… |
| (Be silent for a few minutes two different times) | Facilitator doesn’t probe you to talk | “Tell me what you’re thinking/looking for/doing now?” |
| (Dislike something without providing a reason)
“I don’t like that” or grunt in disgust |
Facilitator doesn’t probe you to ask why | “Why don’t you like that?”
“What is it you don’t like?” |
| (Excessively like or dislike something) | Facilitator shows excessive agreeing | Remain neutral, not too overly friendly like, “mmm hmm” or “OK, thanks.” |
| (Blame yourself for bad UI)
“I must be stupid. I bet everyone else gets this.” |
Facilitator doesn’t reassure | “It isn’t you, just that the UI doesn’t make sense and that’s why we’re here, to look for parts that don’t make sense and fix them.”
“We are learning a lot from you.” |
| (Leave them hanging)
“That’s not what I would call it.” |
Facilitator doesn’t probe you to ask for a better term | “What would be a better term for xxx that makes more sense to you?” |
| “My grandmother would have a hard time with this.” | Facilitator doesn’t ask for their personal opinion | “That’s interesting. How does it work for you personally?” |
| [more difficult]
Talk fast and go way off track of where the task wants you to go |
Facilitator doesn’t gently bring you back on task | “Thank you for that feedback. Can you tell me if you’ve finished task X?” |
You only need to train one person to role-play these difficult participants. Consider running these annually for both new and experienced researchers to practice. You might even want to mix things up and let the inexperienced researcher act as the difficult participant. It will provide them even more empathy for participants, as well as for you.
Finally, try adding your own “challenging” participant personalities to the list. Donna Tedesco and Fiona Tranquada’s The Moderator’s Survival Guide has even more ideas of possible ways things go wrong in a usability test along with what to do about it.
