At AutoTrader.com, we have learned the value of actively involving observers when conducting usability studies. We have always invited stakeholders, such as product and project managers, visual designers, and interaction designers, as well as individuals with periphery interests such as marketing researchers and QA analysts to observe usability studies. However, our practice has evolved to improve our approach and provide a more valuable experience for observers and our usability team alike. The goal of this evolution has been to ensure that observing usability sessions is an effective use of time for our colleagues while also inviting them to provide valuable input for our findings reports. After reading this article, our hope is that you will not only gain insight into the advantages of incorporating observers into your practice, but that you will learn how to do so effectively.
Seeing is Believing
Attending usability study sessions provides observers with a unique opportunity to watch and listen to users as they interact with products. Prior to a study, stakeholders often find it difficult to imagine or articulate issues that may be encountered by end users, and they find it extremely valuable to witness issues unfold. It is during sessions that they get to see their concerns validated or eliminated and new issues and ideas uncovered.
Two (Or More) Heads Are Better Than One
Immediately after study sessions, our team conducts a debriefing where observers have the opportunity to learn from, and discuss findings with, the usability team and fellow observers. The atmosphere is one of energetic collaboration, with a diverse group represented in an open forum. In addition to seeing and hearing users’ perspectives, stakeholders are given the opportunity to see their project through the eyes of other observers, which provides valuable insight. They leave with enhanced understanding of issues and sometimes even with solutions. As the adage “two heads are better than one” suggests, valuable collective discoveries emerge beyond what a single observer might see.
The Magic of Mutualism
Observers are not the only ones to benefit from their inclusion. Researchers also benefit in gaining insights beyond what may have been captured in their session notes and recordings. Discussions with observers help researchers gauge if there is consensus on an issue or, conversely, when an issue may be contentious. Observers sometimes use interesting terminology and phrasing researchers might not be aware of, and key stakeholders oftentimes illuminate the history of a project, along with business considerations and challenges they face. As we describe findings and offer recommendations in the findings report, we’ve discovered that learnings from observer comments can help us craft our research debriefs in a more persuasive manner.
As a direct result of our observer management process, we have seen an increased interest in the work we do from both project stakeholders who are well-versed in our practice, and from observers who had not previously been exposed to our work. This has increased awareness in how we can help projects, giving us the opportunity to undertake wider variety of research activities.
Watching users interact with products is no longer framed as an “if” or “maybe” aspect of a project timeline—it is now framed as “when” and “how.”
Establishing Best Practices
As you can see, the value of inviting others to observe usability sessions has been firmly established in our practice for quite some time. However, the day-to-day logistics of exactly how to include observers effectively has seen significant evolution in our practice. We have learned a number of lessons along the way.
Before the Study: What Doesn’t Work
It used to be that when inviting observers to study sessions, a general email about the study was sent and observers were asked to reply with which sessions they wanted to attend. Our team tracked responses, but formal calendar invites weren’t sent. As you can imagine, observer no-shows were common.
Additionally, when preparing the observation room prior to a study, we did not place note-taking materials at each observer seat. Rather, note-taking materials were placed in a small stack between seats and the test plan and participant demographic sheet were placed at each seat. In most cases, note-taking materials went unnoticed, while observers noticed the test plans and participant demographic sheets immediately.
What Does Work
Our process for inviting observers to study sessions now involves the following steps:
- Outlook calendar invites are created for each session of the study.
- Key stakeholders are automatically invited to each session two weeks prior.
- An email describing the study and welcoming observers is sent out to others who may be interested in the sessions.
- Outlook calendar invites are sent to individuals who responded to the email.
We allow up to ten observers per session, with executives, upper management, and key stakeholders given highest priority. We then fill the remaining observer slots with members of our department and other research teams on a first-come, first-served basis. Observer no-shows have decreased significantly since we started sending calendar invites.
Additionally, from our first contact with potential observers, we now establish and define the observer role so that participation expectations are set. In the invitation email, we state that observers are responsible for recording at least two findings and sharing them with the group after each session. The observer role is emphasized again when the calendar invite is sent to each individual.
When setting up the observation room, we now strategically stack observer materials at each seat in the following order from top to bottom: large Post-It pad with a Sharpie, a cover sheet that lists brief observer “rules,” the study test plan, and the participant demographics list (see Figure 1). We also pre-label a whiteboard that can be used to organize the observer Post-Its after each session.
During Each Study Session: What Doesn’t Work
Initially, to engage observers during study sessions, we tried allowing them to write down anything they found interesting on Post-Its using a Sharpie. We did not give them clear instructions or set expectations as to how the notes would be used. In the end, observers wrote very little.
We also invited observers to be the official note-taker during sessions, but most were not interested. When there was interest, the quality of the notes suffered due to difficulty determining what information was appropriate to record and in what format.
At one point, we asked nothing of observers during sessions and did not provide them with a means to take notes. Consequently, observer engagement suffered.
In general, due to the lack of organization and clear instruction with the above methods, observers were not able to channel their thoughts in a way that was useful. As a result, they often engaged in conversations with one another, creating a noisy environment and making it difficult for the note-taker and other observers to hear participants.
What Does Work
Our current approach is much more structured. In the five minutes before each session, a usability team member verbally reminds observers that their participation is important. Specifically, we emphasize that the Post-Its provide a means for us to consider their input in study reports, and we describe how each observer will be given the opportunity to share their notes.
We also emphasize the purpose of the study and provide examples of findings that are and are not appropriate to capture during a usability study, focusing on observations of user behavior rather than user opinions.
Observers have been notably quieter as they actively and enthusiastically record their observations on the Post-Its, knowing they will have an opportunity to discuss the issues after the session. We have found that giving observers a specific task, providing them with materials to complete the task, and explaining why their contribution is important works wonders at keeping the noise level down and involvement up.
After Each Study Session: What Doesn’t Work
Initially, our session debriefs were short, casual, and unstructured, with some rough notes captured informally on a whiteboard. Sometimes observers turned in handwritten notes, but we didn’t collect them in an organized fashion. Observers often left sessions without key takeaways, or with incorrect takeaways, and missed out on hearing insights from fellow observers because very few observers participated.
To increase collaboration and participation, we formalized our debriefs and recorded organized notes on a whiteboard. Originally, we organized notes by participant and only recorded findings that were unique to a given session. Observers were noticeably quiet during the debriefing or engaged in side conversations since some may not have attended previous sessions and did not know what was unique about the session they attended. Additionally, the notes were of limited use to our team when creating the findings report since we group our formal findings by screen or topic and not by participant.
We eventually began labeling the white board by topic and recorded notes with frequency information next to each finding. This change helped when transferring the notes to the findings report, but it did not increase observer engagement. We inadvertently created awkward, frequent lulls in conversation when turning to the whiteboard to handwrite findings, and ended up talking amongst ourselves more than we would have liked. Observers relied heavily on memory as discussion progressed, sometimes resulting in misremembering an event or discussing issues that were most memorable, while overlooking or downplaying other issues.
What Does Work
We have now evolved into a more structured method that sets expectations around the preferred level of observer involvement and provides clear guidelines for observer participation. We set the stage at the beginning of debriefs by reminding observers that they play an important role, describing the purpose of the debrief and how long it will take, and reminding them that everyone will get to share their notes.
Each observer then shares their Post-It notes with the entire room and helps categorize them on the whiteboard (see Figure 2). Involving the observers in the notes-mapping process assists in keeping the conversation flowing. For example, we might ask an observer, “Where do you think this goes on the board?” If time is running out, we focus on a rough mapping during the debriefing and correct as necessary after the debriefing.
We designate a “parking lot” section on the whiteboard so we can politely acknowledge notes that are off topic. As a general rule of thumb, we set aside at least thirty minutes for debriefs after each session, and debriefing does not end until all observers have shared their notes. At the end of each debriefing, or earlier if needed, we caution our observers not to make generalizations too early and without having observed multiple sessions.
The levels of enthusiasm and energy remain consistent throughout the debriefing as observers take ownership of the documented findings. Lulls in conversation are no longer an issue since we quickly categorize the Post-Its, and observers are eager to empty their hands of Post-Its. Interestingly, as an added bonus, observer debriefing has almost become observer led, requiring less work of our team than any of the prior methods we tried.
Conclusion
We take great pride in having created an atmosphere of transparency and collaboration between our usability team and observers of usability studies. Building this atmosphere has not been without its challenges, but we have successfully found ways to balance observer inclusion with efficient study goal accomplishment.
By openly and directly communicating with observers prior to studies, providing them clear instructions and materials during studies, and giving them an organized way to express themselves after study sessions, your practice will only improve. As your practice improves, never forget to express your appreciation frequently. After all, a rising tide lifts all boats.可用性研究计划应该包括观察员参与方式。实际案例研究表明，提前做一点准备工作有助于您更有效地帮助观察员履行职责。可以获得的好处包括：更好地展示您的工作、更深层次地讨论研究结果和提出的解决方案。了解如何向观察员传授可用性测试中的关注点，并确保每个人都有发言权。
The full article is available only in English.사용성 연구를 계획할때는 관찰자를 참여시키는 여러 가지 방법들을 생각해야 합니다. 한 가지 실제의 사례 연구를 통해서 미리 작업을 좀 해두는 것이 관찰자들을 더욱 효과적으로 참여시키는 데 도움이 될 수 있음을 보여줍니다. 장점으로는 작업의 가시성이 더 높아지며, 발견사항과 제안된 해결책에 대해 좀 더 깊은 논의가 가능하다는 것 등이 포함됩니다. 관찰자들에게 사용성 테스트에서 어떤 점들을 찾아봐야 하는지 가르치고, 모든 사용자가 자신의 의견을 말하도록 하는 방법을 배웁니다.
전체 기사는 영어로만 제공됩니다.A participação dos observadores é uma parte importante da realização de estudos de usabilidade. Um pouco de trabalho prévio pode ajudar a envolver os observadores de forma mais eficaz.
O planejamento de um estudo de usabilidade deve incluir maneiras de envolver os observadores. Um estudo de caso de uma empresa mostra que um pouco de trabalho prévio pode ajudá-lo a envolver os observadores de forma mais eficaz. As vantagens incluem melhor visibilidade para seu trabalho e uma discussão mais profunda sobre as descobertas e soluções propostas. Aprenda como ensinar aos observadores o que procurar em um teste de usabilidade e assegure-se de que todos possam se manifestar.
O artigo completo está disponível somente em inglês.ユーザビリティ活動の計画には、オブザーバーを参加させるための方法が含まれるべきだ。あるプラクティスのケーススタディによれば、事前のちょっとした準備がオブザーバーをより効果的に引き込むのに有効であるとのことである。自らの作業の可視性が向上し、見出された事柄や提案されるソリューションに関する掘り下げたディスカッションのできることなどが、その利点として挙げられる。ユーザビリティテストで観察すべきことをオブザーバーに教え、参加者は誰でも意見を持っているということを学ばせることが大切だ。
原文は英語だけになりますLa participación de observadores es un aspecto importante en el estudio de la usabilidad. Un poco de trabajo por anticipado puede promover una participación más efectiva de los observadores.
La planificación de un estudio de usabilidad debe incluir formas de involucrar a los observadores. Un estudio de casos de una práctica permite demostrar que un poco de trabajo por anticipado puede promover una participación más efectiva de los observadores. Las ventajas incluyen una mejor visibilidad en su labor y un análisis más exhaustivo de los resultados y las soluciones propuestas. Aprenda cómo enseñar a los observadores qué deben buscar en un examen de usabilidad y garantizar que todos puedan expresar su opinión.
La versión completa de este artículo está sólo disponible en inglés
