Keeping a diary is not just a hobby undertaken by teenage girls trying to make sense of life and love. Diary methods have been widely used in medical research where patients are asked to keep a diary during a clinical trial or disease treatment. In Human-Computer Interaction, a diary study is a qualitative technique for collecting data on what users have done or experienced. Much like a travel journal contains descriptions of the traveler’s experiences, a UX diary contains descriptions of the user’s experiences with a system.
What kind of UX research questions could diaries help answer? Depending on the design of your diary study, you may get information on the overall impression of a specific device, usage of features, technological acceptance, emotions associated with task performance, or learnability of an application. Of course you could also get this information with a survey, an interview, or a user test. But only diary methods will give you access to temporal and longitudinal information gathered in a natural context of the interaction.
Advantages of Diary Methods
Studying Temporal Dynamics
As stated in “The User Experience White Paper,” there are several time spans of user experience depending on the moment of usage (see Figure 1). Typical methods used to study UX shed light onto only one kind or moment of the UX. Surveys are often related to Episodic UX: users are asked to assess the interaction after usage. User testing helps collect data on Momentary UX. You may also have an idea of Anticipated UX by conducting focus groups.
The main advantage of a diary study is that it allows collecting longitudinal information. Let’s imagine you would like to assess a new mobile application. The first part of the diary study may be administered prior to usage by asking users how they imagine the application, what they would expect from it, and if they have any prior experience with similar applications. Then, you might ask them to report their everyday impressions and feelings during usage. Finally, during a debriefing interview or a post-study questionnaire, you may ask the users to reflect on their experience after usage.
Reporting Events and Experiences in Context
Another issue we should be always mindful of is the impact of context on the results. For example, when conducting a user test or an interview, the presence of the researcher may bias the results, leading to a positive evaluation of a system despite poor user performance. This is called a social desirability bias. When we know we are being watched, we will tend to behave in a way we believe is socially acceptable or desirable.
As researchers, we should want to capture life as it is and not how it looks in a controlled setting. The emergence of mobile technology especially highlights this need to study use “in the field” (see Figure 2). That’s when the diary methods come in handy, as they focus on reporting events and experiences in their natural context.
My company developed a collaborative environment to address the needs of architects and engineers in the construction sector. The mobile application was tested in a usability lab, and flaws were identified and fixed. But once used on a construction site, there were more problems than expected. Effects of dust, noise, brightness, and handling were underestimated during both the design and evaluation steps. Moreover, collaborative issues in this real on-the-go setting were quite different than those simulated. The subsequent use of a diary study helped improve the system by revealing real user issues and needs in the context of their workplace.
Determining the Antecedents, Correlates, and Consequences of Daily Experiences
As mentioned before, UX is rooted in different time spans, each one being influenced by another. Momentary UX is colored by Anticipated UX in the way that user’s expectations have to be met or exceeded to lead to a positive experience. Then, Momentary UX will be distorted by the filter of cognitive processes when changing into Episodic UX and Cumulative UX. Expectations, mindsets, moods, and social or physical contexts all color user experiences. A diary study captures these influences, shedding light onto how the UX in each time span has formed.
For example, findings from diary studies can reveal that it was the bad mood of the user or brand criticism heard on TV that impacted the user’s assessment of the interface. You may also discover that the game you are evaluating provides enjoyment only when the user interacts with close friends or family, even if the functionality supporting this was neglected by designers. Finally, you may notice that a single flaw you considered as minor discourages the user and leads to system abandonment.
Limitations of Diary Studies
Nothing is perfect! As with most methods, you need to look at the pros and cons of designing a diary study. The main disadvantages of the method are the cost and time associated with:
- Participant recruitment. The quality of the results depends on the participants since the diary is bound to the expressive ability of the writer. This is especially true for a study with a lot of open-ended questions. You also have to achieve a high level of participant commitment to obtain sufficient and reliable diary entries.
- Training or briefing sessions. A diary study often requires detailed training sessions to ensure that participants fully understand what needs to be reported, how, and when.
- Data analysis. Analyzing diary entries is time consuming. This activity takes even more time in the case of a pen-and-paper diary.
Designing a Diary Study
Eliciting Diary Entries
There are three categories of diary protocols based on the way in which entries are elicited:
- Interval-contingent protocol, in which participants have to report their experience at regular predetermined intervals (for example, every two hours or every day).
- Signal-contingent protocol, which uses a signaling device to prompt the participants to make an entry.
- Event-contingent protocol, which requires participants to report each time a specific event occurs.
Protocol selection depends on the purpose of the study and activities under investigation. The interval-contingent option, although widely used, does not help participants remember to make diary entries, which may lead to missing data. Signal-contingent protocols overcome this drawback but are intrusive. An event-contingent protocol might be used especially when you are interested in studying the experience created by a specific event, such as a system error or notification. Rate and timing of self-report have to be set up according to research needs. Do not be too demanding or your diary will become a burden to your participants; a maximum of two to three entries per day should be enough.
Involving Users
One of the major challenges in diary studies is maintaining participant commitment at a high level to ensure diary completion. You have to make sure that participants understand the scope and descriptive depth needed for the diary entries. During the study, you can remind participants of the importance of diary completion. A cash or gift incentive, or the possibility of keeping the assessed device, can be very helpful. However, be careful not to base the incentives on data quantity or you may get tangled up in irrelevant information.
Pen and Paper vs. Electronic Diaries
Because UX studies are often related to the use of technological devices, the choice of an electronic report is often judicious. It simplifies data analysis and gives the opportunity to combine diary entries and log files (for example, date, time, and task performed). E-diaries also provide participants with innovative forms of reporting. Imagine a diary study through voicemail entries, text messages, video, or pictures!
However, you should always try to adjust the method to the target users. If you are conducting a study with elderly people, a pen and paper diary will probably be more appropriate. Your participants should feel comfortable with the process of keeping a diary.
Diary Structure
Diaries may be open format (users record activities and events in their own words) or highly structured (where closed-ended questions are pre-categorized). Do you wish to encourage general reflection or need precise information? A mixed approach is often adopted to collect both qualitative and quantitative data. Do not forget to add a clear set of instructions on how to complete the diary, and stress the importance of recording events as soon as possible after they occur. You may also use non-verbal tools to facilitate completion (see Figure 4), especially for specific target users such as children or people with a disability.
Collecting and Analyzing Data
After designing your diary, do not forget to pilot-test it before the study launches. Participants should keep the diary for a maximum of two weeks, as people’s commitment quickly decreases over time. If the diary is electronic, you should monitor its completion for each participant. Do not hesitate to contact participants during the study to ensure that everything is going well.
At the end of the study, conduct follow-up interviews with your participants. That’s your opportunity to have them explain some of their entries. A good understanding of participants’ qualitative answers is required to correctly interpret the data.
The analysis of diary entries depends on how structured the diary is. Quantitative data may be analyzed using any statistical analysis software. Dealing with qualitative data is more challenging but certainly worth the effort because these will provide you with valuable information on subjective experience.
Example of a Before-Usage Diary Study
A diary study of rendezvousing (described in the Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work) examined how people meet up, with the goal of understanding how technology could support them.
Martin Colbert used a diary method because the events of interest were “too rare, private, and geographically dispersed for direct observation, and users too easily forget important details of their behavior to report them accurately long after the event.” Participants had to report their rendezvousing behavior for fourteen days. One diary entry was made for each rendezvous event. Each entry was composed of an open-ended, narrative description in the participant’s own words of what happened and why, and the participant’s responses to a rendezvous questionnaire, which asked for specific details of each event (see Figure 3).
The researchers used the results to extract the overall structure of rendezvousing and to build typical rendezvousing scenarios. Eight possible causes of problematic rendezvous were identified, including transportation problems, over-running of previous activities, poor planning, lack of travel information, lack of geographical information, lack of information about other rendezvousers, spontaneous additional tasks, and failure to value success.
The study helped identify what kind of position-aware service was likely to be most useful and how it could be designed to shape rendezvous experiences.
Quantifying UX Over Time
I have used the abridged AttrakDiff tool (www.attrakdiff.de) to evaluate the hedonic and pragmatic quality of a mobile application (see Figure 5). By asking participants to report their feelings and impressions using a short survey, I was able to draw a UX chart representing the evolution of UX over time (see Figure 6).
The Snippet Technique
The snippet technique developed by the Stanford HCI Group aims at overcoming the limitations of diary studies performed under mobile or active conditions, where there is no time to make thorough entries. Participants capture text, MMS, or voice snippets when on the go and later annotate the snippets on the web (see Figure 7).
用日志方法来研究用户体验具有很多优势。本文介绍为什么要使用日志以及如何设计一个日志研究。作者对最适合采用这些方法的研究问题、可以根据日志研究的发现做出的设计决策、进行研究所需的工具，以及分析日志报告的方法提供了实用指导。
The full article is available only in English.UX를 연구하고 평가하는 것은 경험 있는 전문가를 위해서도 어려운 과제입니다. 슈퍼히어로가 되지 않거나 수정 구슬 없이 UX와 관련된 모든 요소를 염두에 두고 사용자의 관점으로부터 경험을 얻어낼 수 있는 방법을 어떻게 찾아낼 수 있을까요?
UX 연구를 위한 일기 형식 방법의 장점은 상당히 많으며 본 논문은 왜 당신이 연구를 위해 일기를 이용해야 하며 어떻게 일기 형식 연구를 설계하는지 설명합니다. 저자는 이 방법에 가장 잘 맞는 연구 문제, 일기 형식 연구의 연구 결과를 바탕으로 할 수 있는 디자인 결정, 연구 수행에 필요한 도구 및 일기 형식 리포트 분석 방법 등에 관한 실용적인 지침을 제공합니다.
The full article is available only in English.O estudo e a avaliação da experiência do usuário é uma tarefa desafiadora, mesmo para profissionais experientes. Sem ser um super herói ou ter uma bola de cristal, como podemos encontrar uma metodologia capaz de capturar a experiência do ponto de vista do usuário, levando em conta todos os fatores relacionados à experiência do usuário?
As vantagens dos métodos que utilizam diários para o estudo da experiência do usuário são numerosas e este artigo descreve por que você deve utilizar diários em sua pesquisa e como projetar um estudo utilizando diário. O autor fornece uma diretriz prática com relação a questões de pesquisa mais adequadas para esses métodos, decisões de projeto que podem ser tomadas com base nos resultados de um estudo utilizando diário, ferramentas necessárias para conduzir a pesquisa e formas de analisar os relatórios do diário.
O artigo completo está disponível somente em inglês.UXの調査および評価は経験豊富なプロでも容易なことではない。スーパーヒーローであったり、魔法の水晶玉を持たない限り、どうすれば、UXに関連したすべての要因を考慮しつつ、ユーザ視点からの体験を把握できる調査方法を見つけ出せるだろうか。
UXの調査における各種ダイアリー法のメリットはいくつも挙げられるが、この記事ではなぜ研究においてダイアリーを使うべきなのか、またダイアリー調査をどのようにデザインすべきなのかについて説明する。筆者は、これらの調査方法に最適なリサーチクエスチョン、ダイアリー調査での結果をもとに行えるデザイン決定、調査を実施するにあたって必要なツール、ダイアリーのレポートの分析方法について、実用的なガイダンスを提供している。
The full article is available only in English.Las ventajas de los métodos que utilizan diarios para el estudio de UX son numerosas y en este artículo se describe por qué debería utilizar diarios en sus investigaciones y cómo diseñar un estudio que incluya diarios. La autora ofrece consejos prácticos respecto de las preguntas más adecuadas en estos métodos de investigación, las decisiones de diseño que se pueden tomar en base a los resultados de un estudio con diarios, las herramientas necesarias para realizar la investigación y las formas de analizar los informes que incluyen diarios.
La versión completa de este artículo está sólo disponible en inglés.
