Search Results for: diary studies

Little Red Sprite Learns Finance: An Exploration of Mobile Diary Studies Using Avatars and Chat Apps

May 21, 2017 No Comments

In this article, we share considerations and lessons learned from using WeChat as a mobile diary that can be applied to future studies using chat apps.

A mobile screen with the login for WeChat — Figure1. WeChat as a mobile diary.

The mobile diary is a great remote research tool to help uncover the daily lives of consumers and gather field-based consumer insights, especially as naturalistic data can be difficult to obtain through focus groups, interviews, or observations. This is particularly true when presented with geographical and temporal constraints. In a typical mobile diary study, participants record their reactions and responses to the subject matter (for example, brand communication, shopping activity) while on-the-go, on their own mobile devices, the very moment they experience them.

Our Singapore-based research team used an existing chat app, WeChat, as a diary tool during a series of mobile diary studies that focused on understanding how Chinese consumers go about their banking and financial activities in China. Use of an avatar rather than conventional human facilitation allowed us to engage more authentically with consumers and gather richer insights.

Finding the Right Tool

If we disregard China as the context (think firewall, app usage restrictions, and language barriers), there are a plethora of mobile diary tools available, ranging from paid to free, from custom apps to good old email. Custom mobile diary apps like dscout, ethos, and indeemo can be useful in research as they not only help collect information but also allow basic coding/tagging or grouping, enabling researchers to see data clearly.

Collaborative tools like Evernote and Google Docs also enable researchers to receive real-time updates from participants as they record their activity in text, photos, and/or audio.

However, for our study we wanted to leverage existing behaviors of how people communicated with loved ones, akin to documenting their lives as they chatted with their friends and families. This led us to considering chat apps as a tool. We wanted to avoid having participants purchase or experience potential difficulties using a new mobile app, especially since our participants lived within a different culture and used a different language than us.

Reducing the time needed for participants to grasp the technicalities of how to use a mobile diary study tool allowed us to devote more time to explain the value of the research and encourage their buy-in. Based on our previous experience, participants were willing to be more open, and, in turn, provide more valuable insights when they realized their recorded financial experiences could help banks provide them better services.

WeChat was selected for its ubiquity in China, with a reported 750-plus million users monthly, allowing us to benefit from participants’ familiarity. Other popular chat apps in Asia like Google Talk, Facebook Messenger, WhatsApp, Korea’s KakaoTalk, and Japan’s Line Messenger were disqualified due to China’s firewall and data restrictions.

In addition to chatting about their financial behavior using text, participants were encouraged to use pictures and voice messages. Some preferred recording audio while on the go, especially some older participants who seemed to be more comfortable with voice messages. Voice messages proved to be useful in that we could hear more contextual data, too. For example, we could hear the change in tone of voice if the participant was in a hurry or a hushed voice if speaking at home with a baby present.

Although the name WeChat suggests a chat focus, it also has a social media feature, allowing users to share posts with selected friends on their individual walls. Gaining access to users’ social media walls after participants added our WeChat account as a “friend” provided researchers a sense of overall lifestyle, allowing for cross-referencing wall postings with data collected from their diaries. Similarly, participants could refer to the objective of the study and download related content from the researcher’s wall.

Calendar even screen (in Chinese) — Figure 2. Gaining access to users’ WeChat walls provided researchers a deep view of the lifestyle and behavior of participants, allowing for cross-referencing with data collected from their diaries.

Calendar event screen (in Chinese) — Figure 3. Participants found information on the objective of the study and downloaded related content from the researchers’ WeChat wall.

Lastly, though not an initial consideration in selecting WeChat as the medium, the payment features on WeChat largely aided participants in understanding the nature of our study. Following the success of WeChat Red Packets (微信红包), which mimics the Chinese tradition of gifting money in physical red envelopes called hongbao (红包), Tencent has put considerable focus into WeChat Pay (微信支付), which can be used to transfer money between WeChat users (peer-to-peer) and make payments online and with participating offline retailers.

Leveraging familiarity (participants were all WeChat users), we referenced WeChat Pay features as examples of financial activities to record and discuss, on top of other common examples like visiting a bank or paying for groceries.

Screens in Chinese as described in the caption — Figure 4. Screenshots from participants using payment features on WeChat. From left to right, WeChat Pay for taxi services, WeChat Pay for merchant purchases, a WeChat Red Packet message, and a WeChat Red Packet containing digital money as a gift.

Someone to Talk To: Avatar as Researcher

Our initial challenge with the mobile diary was that participants assumed the diary facilitator understood the nuances underlying the recorded activity and bypassed cultural explanations. For example, a participant posted, “I spent 47yuan on taxi today,” but stopped short of explaining how exactly the payment was made. It was only via post-study interviews that we found taxis are commonly hired via mobile platforms such as DiDi in China, instead of flagging one off the street. Another challenge was that many were protective about personal financial habits or sharing such information with a stranger-researcher from a foreign land.

Given the social and conversational nature of WeChat, we turned to designing an agent to act as our face—our representative and voice. For our purposes, such a character needed to be friendly, approachable, culturally appropriate, and act like a curious outsider.

Two cartoon-y icons — Figure 5. Alibaba’s Taobao (online shopping site) Mascot 淘公仔 Tao Doll (left) and Tencent’s QQ (mobile messaging platform) Mascot QQ企鵝 QQ Penguin (right).

At the beginning of each study we introduced a curious cartoonish alien we named Little Red Sprite (小红精灵) wanting to understand ‘Earthlings’ to encourage participants to document in more detail, explaining their activities and all relevant Chinese cultural aspects–just as they would do for an interested non-local. Along with designing the agent to look endearing, we evaluated the mascots of China’s Internet giants and made Little Red Sprite’s conversation with participants more colloquial and playful.

A screen with the mascot and a message in Chinese — Figure 6. Our mascot, Little Red Sprite, saying hello and explaining the mobile diary study.

Chatting Is What a Chat App Is About

Each study was 30 days long; with time participants’ posts started to be simpler, less frequent, and questions like, “Am I doing this right?” began appearing. We learned how crucial it was to close the loop while verifying our interpretations. To elicit a higher quality and quantity of posts, we leveraged WeChat’s sticker culture to engage in dialogue and reply to posts with customized stickers to appear more personable. The role of Little Red Sprite was critical here to prevent participants from being intimidated or overwhelmed by the requirements of the research. We believed an advantage of this type of avatar was in its social capability, essential to the continued dialogue. Care was taken to have Little Red Sprite display the emotions of gratitude and curiosity, as opposed to a tone of disappointment when posts were not made.

Also, participants were interviewed directly on WeChat posts, such as, “You mentioned heading to the supermarket with your credit card. Were there specific benefits of using a credit card?” Or, “I noticed your posts hardly mention shopping online. Do you usually shop online?” Dialogue not only demonstrated the kind of reciprocity that was inherently motivating, but also helped in drawing a detailed picture of the activity.

The relationships built up over the course of the diary study were also essential to the success of follow-up interviews conducted. The trust and familiarity gained by participants enabled intimate sharing as opposed to a rigid Q&A. Dialogue with participants on WeChat helped to scope questions for the follow-up interview phase, prompting in-depth questions like, “Could you explain why you have different credit cards and bank accounts for shopping online and shopping offline?”

Little Red Sprite, the agent, seemed to resonate well with participants and their responses were open and friendly, especially during the follow-up interviews. Participants addressed the researchers as “Little Red Sprite” and apologized through WeChat for not spending money or not posting. These were responses we had never received from previous (non-agent) studies! Their posts also had more explanations and included participants’ comments like, “Oh yes, you might not know, but in China giving your partner access to your bank account is a sign of trust. It’s common for girls to handle her boyfriend’s finances.”

Conclusion

Financial studies are often tricky because they involve sensitive data. Not everyone is open to sharing how much they spend, what they spend on, or how much they earn.

While our study was designed for the Chinese market and used a chat app locals were familiar with, the same principles can be applied to other markets and cultures where there is a high penetration of social chatting apps.

Participants’ understanding of how their responses were of value encouraged them to contribute more expressively and regularly. Enticing participants to continuously post about their day required vigilance and creativity in both feedback and encouragement.

The overall success of the technique was measured by the higher quality data (the amount of detailed and personal information) provided by participants, and by engagement (how frequently the participants posted). For some, posting even continued after receiving notice that the study had ended. The relationships developed (with Little Red Sprite) led to the success of the interviews, as trust and familiarity enabled more open, rich, and detailed conversations.

The study was made possible by NCR and we’d like to thank all of our research participants.

[greybox]

Translated Transcript from Figure 6

Hi, I’m Little Red Sprite from Planet Ingot and I currently work at the customer experience department. I discovered a book about Earth in the library the other day and saw that there was a country called China that’s filled with lots and lots of people. I was so excited and I’m sure there’s a lot to learn about how people on Earth experience banking. Unfortunately, I couldn’t find much information on my own, could you help me understand how you bank? I would like to learn and introduce better ways of banking back to Planet Ingot. Thanks for helping!

[/greybox]

Dear Diary: Using Diaries to Study User Experience

August 15, 2012 No Comments

Keeping a diary is not just a hobby undertaken by teenage girls trying to make sense of life and love. Diary methods have been widely used in medical research where patients are asked to keep a diary during a clinical trial or disease treatment. In Human-Computer Interaction, a diary study is a qualitative technique for collecting data on what users have done or experienced. Much like a travel journal contains descriptions of the traveler’s experiences, a UX diary contains descriptions of the user’s experiences with a system.

What kind of UX research questions could diaries help answer? Depending on the design of your diary study, you may get information on the overall impression of a specific device, usage of features, technological acceptance, emotions associated with task performance, or learnability of an application. Of course you could also get this information with a survey, an interview, or a user test. But only diary methods will give you access to temporal and longitudinal information gathered in a natural context of the interaction.

Advantages of Diary Methods

Studying Temporal Dynamics

As stated in “The User Experience White Paper,” there are several time spans of user experience depending on the moment of usage (see Figure 1). Typical methods used to study UX shed light onto only one kind or moment of the UX. Surveys are often related to Episodic UX: users are asked to assess the interaction after usage. User testing helps collect data on Momentary UX. You may also have an idea of Anticipated UX by conducting focus groups.

Figure 1. Time spans of user experience. (Source: “The User Experience White Paper,” Dagstuhl Seminar on Demarcating User Experience, 2010)

The main advantage of a diary study is that it allows collecting longitudinal information. Let’s imagine you would like to assess a new mobile application. The first part of the diary study may be administered prior to usage by asking users how they imagine the application, what they would expect from it, and if they have any prior experience with similar applications. Then, you might ask them to report their everyday impressions and feelings during usage. Finally, during a debriefing interview or a post-study questionnaire, you may ask the users to reflect on their experience after usage.

Reporting Events and Experiences in Context

Another issue we should be always mindful of is the impact of context on the results. For example, when conducting a user test or an interview, the presence of the researcher may bias the results, leading to a positive evaluation of a system despite poor user performance. This is called a social desirability bias. When we know we are being watched, we will tend to behave in a way we believe is socially acceptable or desirable.

As researchers, we should want to capture life as it is and not how it looks in a controlled setting. The emergence of mobile technology especially highlights this need to study use “in the field” (see Figure 2). That’s when the diary methods come in handy, as they focus on reporting events and experiences in their natural context.

Two photographs of people using a mobile phone: in an office and on a construction site. — Figure 2. On the left: A participant interacting with a mobile application in a usability lab. On the right: A participant using the same application, but in a natural environment. Do you really think that the user experience is the same in both situations?

My company developed a collaborative environment to address the needs of architects and engineers in the construction sector. The mobile application was tested in a usability lab, and flaws were identified and fixed. But once used on a construction site, there were more problems than expected. Effects of dust, noise, brightness, and handling were underestimated during both the design and evaluation steps. Moreover, collaborative issues in this real on-the-go setting were quite different than those simulated. The subsequent use of a diary study helped improve the system by revealing real user issues and needs in the context of their workplace.

Determining the Antecedents, Correlates, and Consequences of Daily Experiences

As mentioned before, UX is rooted in different time spans, each one being influenced by another. Momentary UX is colored by Anticipated UX in the way that user’s expectations have to be met or exceeded to lead to a positive experience. Then, Momentary UX will be distorted by the filter of cognitive processes when changing into Episodic UX and Cumulative UX. Expectations, mindsets, moods, and social or physical contexts all color user experiences. A diary study captures these influences, shedding light onto how the UX in each time span has formed.

For example, findings from diary studies can reveal that it was the bad mood of the user or brand criticism heard on TV that impacted the user’s assessment of the interface. You may also discover that the game you are evaluating provides enjoyment only when the user interacts with close friends or family, even if the functionality supporting this was neglected by designers. Finally, you may notice that a single flaw you considered as minor discourages the user and leads to system abandonment.

Limitations of Diary Studies

Nothing is perfect! As with most methods, you need to look at the pros and cons of designing a diary study. The main disadvantages of the method are the cost and time associated with:

Participant recruitment. The quality of the results depends on the participants since the diary is bound to the expressive ability of the writer. This is especially true for a study with a lot of open-ended questions. You also have to achieve a high level of participant commitment to obtain sufficient and reliable diary entries.
Training or briefing sessions. A diary study often requires detailed training sessions to ensure that participants fully understand what needs to be reported, how, and when.
Data analysis. Analyzing diary entries is time consuming. This activity takes even more time in the case of a pen-and-paper diary.

Designing a Diary Study

Eliciting Diary Entries

There are three categories of diary protocols based on the way in which entries are elicited:

Interval-contingent protocol, in which participants have to report their experience at regular predetermined intervals (for example, every two hours or every day).
Signal-contingent protocol, which uses a signaling device to prompt the participants to make an entry.
Event-contingent protocol, which requires participants to report each time a specific event occurs.

Protocol selection depends on the purpose of the study and activities under investigation. The interval-contingent option, although widely used, does not help participants remember to make diary entries, which may lead to missing data. Signal-contingent protocols overcome this drawback but are intrusive. An event-contingent protocol might be used especially when you are interested in studying the experience created by a specific event, such as a system error or notification. Rate and timing of self-report have to be set up according to research needs. Do not be too demanding or your diary will become a burden to your participants; a maximum of two to three entries per day should be enough.

Involving Users

One of the major challenges in diary studies is maintaining participant commitment at a high level to ensure diary completion. You have to make sure that participants understand the scope and descriptive depth needed for the diary entries. During the study, you can remind participants of the importance of diary completion. A cash or gift incentive, or the possibility of keeping the assessed device, can be very helpful. However, be careful not to base the incentives on data quantity or you may get tangled up in irrelevant information.

Pen and Paper vs. Electronic Diaries

Because UX studies are often related to the use of technological devices, the choice of an electronic report is often judicious. It simplifies data analysis and gives the opportunity to combine diary entries and log files (for example, date, time, and task performed). E-diaries also provide participants with innovative forms of reporting. Imagine a diary study through voicemail entries, text messages, video, or pictures!

However, you should always try to adjust the method to the target users. If you are conducting a study with elderly people, a pen and paper diary will probably be more appropriate. Your participants should feel comfortable with the process of keeping a diary.

Diary Structure

Diaries may be open format (users record activities and events in their own words) or highly structured (where closed-ended questions are pre-categorized). Do you wish to encourage general reflection or need precise information? A mixed approach is often adopted to collect both qualitative and quantitative data. Do not forget to add a clear set of instructions on how to complete the diary, and stress the importance of recording events as soon as possible after they occur. You may also use non-verbal tools to facilitate completion (see Figure 4), especially for specific target users such as children or people with a disability.

A series of 5 cartoon faces indicating a range of emotions through the mouth (turned down to turned up) and other features. — Figure 4. The Self-Assessment Manikin is a non-verbal scale to measure emotions.

Collecting and Analyzing Data

After designing your diary, do not forget to pilot-test it before the study launches. Participants should keep the diary for a maximum of two weeks, as people’s commitment quickly decreases over time. If the diary is electronic, you should monitor its completion for each participant. Do not hesitate to contact participants during the study to ensure that everything is going well.
At the end of the study, conduct follow-up interviews with your participants. That’s your opportunity to have them explain some of their entries. A good understanding of participants’ qualitative answers is required to correctly interpret the data.
The analysis of diary entries depends on how structured the diary is. Quantitative data may be analyzed using any statistical analysis software. Dealing with qualitative data is more challenging but certainly worth the effort because these will provide you with valuable information on subjective experience.

Example of a Before-Usage Diary Study

A diary study of rendezvousing (described in the Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work) examined how people meet up, with the goal of understanding how technology could support them.
Martin Colbert used a diary method because the events of interest were “too rare, private, and geographically dispersed for direct observation, and users too easily forget important details of their behavior to report them accurately long after the event.” Participants had to report their rendezvousing behavior for fourteen days. One diary entry was made for each rendezvous event. Each entry was composed of an open-ended, narrative description in the participant’s own words of what happened and why, and the participant’s responses to a rendezvous questionnaire, which asked for specific details of each event (see Figure 3).

View of the form — Figure 3. A diary entry participants were supposed to complete following each rendezvous.

The researchers used the results to extract the overall structure of rendezvousing and to build typical rendezvousing scenarios. Eight possible causes of problematic rendezvous were identified, including transportation problems, over-running of previous activities, poor planning, lack of travel information, lack of geographical information, lack of information about other rendezvousers, spontaneous additional tasks, and failure to value success.

The study helped identify what kind of position-aware service was likely to be most useful and how it could be designed to shape rendezvous experiences.

Quantifying UX Over Time

I have used the abridged AttrakDiff tool (www.attrakdiff.de) to evaluate the hedonic and pragmatic quality of a mobile application (see Figure 5). By asking participants to report their feelings and impressions using a short survey, I was able to draw a UX chart representing the evolution of UX over time (see Figure 6).

Mobile screen with survey — Figure 5. AttrakDiff survey on a mobile phone.

Graph showing change in scores over time. — Figure 6. AttrakDiff scores illustrating how the UX of a mobile application changed over a period of ten days.

The Snippet Technique

The snippet technique developed by the Stanford HCI Group aims at overcoming the limitations of diary studies performed under mobile or active conditions, where there is no time to make thorough entries. Participants capture text, MMS, or voice snippets when on the go and later annotate the snippets on the web (see Figure 7).

Figure 7. Snippets captured during the day serve as memory aides for full diary entries in the evening.[

即使对于有经验的从业人员，研究和评估用户体验也是一项有挑战性的工作。我们不是超级英雄，也没有水晶球，怎样才能找到一种方法，通过考虑所有与用户体验相关的因素，来获得站在用户角度的体验感受？

用日志方法来研究用户体验具有很多优势。本文介绍为什么要使用日志以及如何设计一个日志研究。作者对最适合采用这些方法的研究问题、可以根据日志研究的发现做出的设计决策、进行研究所需的工具，以及分析日志报告的方法提供了实用指导。

The full article is available only in English.UX를 연구하고 평가하는 것은 경험 있는 전문가를 위해서도 어려운 과제입니다. 슈퍼히어로가 되지 않거나 수정 구슬 없이 UX와 관련된 모든 요소를 염두에 두고 사용자의 관점으로부터 경험을 얻어낼 수 있는 방법을 어떻게 찾아낼 수 있을까요?

UX 연구를 위한 일기 형식 방법의 장점은 상당히 많으며 본 논문은 왜 당신이 연구를 위해 일기를 이용해야 하며 어떻게 일기 형식 연구를 설계하는지 설명합니다. 저자는 이 방법에 가장 잘 맞는 연구 문제, 일기 형식 연구의 연구 결과를 바탕으로 할 수 있는 디자인 결정, 연구 수행에 필요한 도구 및 일기 형식 리포트 분석 방법 등에 관한 실용적인 지침을 제공합니다.

The full article is available only in English.O estudo e a avaliação da experiência do usuário é uma tarefa desafiadora, mesmo para profissionais experientes. Sem ser um super herói ou ter uma bola de cristal, como podemos encontrar uma metodologia capaz de capturar a experiência do ponto de vista do usuário, levando em conta todos os fatores relacionados à experiência do usuário?

As vantagens dos métodos que utilizam diários para o estudo da experiência do usuário são numerosas e este artigo descreve por que você deve utilizar diários em sua pesquisa e como projetar um estudo utilizando diário. O autor fornece uma diretriz prática com relação a questões de pesquisa mais adequadas para esses métodos, decisões de projeto que podem ser tomadas com base nos resultados de um estudo utilizando diário, ferramentas necessárias para conduzir a pesquisa e formas de analisar os relatórios do diário.

O artigo completo está disponível somente em inglês.UXの調査および評価は経験豊富なプロでも容易なことではない。スーパーヒーローであったり、魔法の水晶玉を持たない限り、どうすれば、UXに関連したすべての要因を考慮しつつ、ユーザ視点からの体験を把握できる調査方法を見つけ出せるだろうか。

UXの調査における各種ダイアリー法のメリットはいくつも挙げられるが、この記事ではなぜ研究においてダイアリーを使うべきなのか、またダイアリー調査をどのようにデザインすべきなのかについて説明する。筆者は、これらの調査方法に最適なリサーチクエスチョン、ダイアリー調査での結果をもとに行えるデザイン決定、調査を実施するにあたって必要なツール、ダイアリーのレポートの分析方法について、実用的なガイダンスを提供している。

The full article is available only in English.Las ventajas de los métodos que utilizan diarios para el estudio de UX son numerosas y en este artículo se describe por qué debería utilizar diarios en sus investigaciones y cómo diseñar un estudio que incluya diarios. La autora ofrece consejos prácticos respecto de las preguntas más adecuadas en estos métodos de investigación, las decisiones de diseño que se pueden tomar en base a los resultados de un estudio con diarios, las herramientas necesarias para realizar la investigación y las formas de analizar los informes que incluyen diarios.

La versión completa de este artículo está sólo disponible en inglés.

Dear Diary: Getting the Most from Respondent Journals and Diaries

September 12, 2015 No Comments

During a recent study, I asked respondents about their snacking: when and why. You would have thought that these respondents were among the healthiest on the planet. Why, the apples they consumed could have denuded an entire orchard! Luckily, though, I had asked the respondents to keep a mobile diary of their snacking behavior for the week prior to the focus group. How quickly these respondents had forgotten the chips, Twinkies and, yes, even the Philly cheesesteaks that comprised their snacking behavior over the past week. The client received rich insights into aspiration vs. reality, and I had a lot of fun in the process.

Diary research can lead to deeper insights from your respondents. But creating a great diary assignment is more than just passing out blank paper and having respondents note every time they make a cup of coffee or use their online banking app.

What Is a Diary?

Diaries have three basic elements that, taken together, separate them from all other qualitative data:

They are respondent generated.
They capture a specific behavior, activity, experience or feeling.
They happen over time.

Diaries can be given as homework (see Figure 1), but they do not have to be assigned prior to other research. They can work great as a standalone exercise or when assigned after an initial phase of research.

A Word about Nomenclature

In my work, I sometimes use the words “diary” and “journal” interchangeably. In his book Let’s Explore Diabetes with Owls, writer David Sedaris — himself a prolific diarist — noted, “Often, the terms are used interchangeably, though I’ve never understood why. Both have the word ‘day’ at their root but a journal, in my opinion, is a repository of ideas, your brain on the page. A diary, by contrast, is your heart.”

Indeed, there is something very personal conveyed by the term “diary,” which may be exactly what your study needs. If, however, you are asking school-age boys to keep some kind of diary, consider calling it a “journal” instead. I found from personal experience in my own house that “diaries” tend to elicit eye rolls in this demographic.

Why Use Diaries?

Diaries allow us to extend the amount of time that we spend with respondents. Furthermore, diaries allow us access to respondents at times when we might not want to be with them or when the researcher’s presence might compromise the respondent’s ability to be candid.

Diaries help researchers explore three broad types of question:

Usage: Often, a marketer wants to know how the company’s product or service is actually being utilized in the real world, to identify problems or opportunities. Diaries kept over time allow respondents to capture their actual experiences with the client’s product in detail.
Process: Diaries are great at reflecting consumers as they go through a particular experience, and they often provide deep insights about the purchase process. Consider asking your respondents to keep a diary when they buy a car, apply for college or go through chemotherapy and to note the expected and unexpected steps they take along the way.
Experience: Sometimes, marketers have broader questions that they need to answer to help drive innovation or marketing messages. Experiential diaries help answer the question, ‘What is it like to…?”

Ultimately, diary studies are powerful for clients because they tend to lead to previously undiscovered consumer needs, which then point to opportunities for clients to innovate (See Figure 2).

Which medium Is best?

Believe it or not, paper can be a fabulous medium for the right type of diary study. Digital, of course, is rapidly expanding our ability to be with consumers 24/7, so it is important to think about what best fits your study.

Paper

Paper diaries provide a wealth of opportunity to ask respondents to think and work creatively. For example, a paper diary can allow respondents to write, draw and illustrate in ways they would struggle to do with a digital medium. A major drawback of paper diaries, however, is that you do not get to see what a respondent is working on in real time. If there is a problem with your instructions or if you wanted diary data to help drive discussion-guide development, you are out of luck.

Digital

More and more, the smartphone is becoming the digital-diary medium of choice; however, respondents can also participate in digital diaries using their PC or tablet. The pervasiveness of digital tools makes them particularly attractive for diary exercises: Since most of your respondents are probably carrying their mobile phones with them at all times, it makes good sense to ask them to use those devices to complete a diary exercise. That said, mobile technology tends to favor shorter, less introspective responses than one might get with, say, paper or a diary that uses a PC-based platform where respondents can type at length.

Table 1. Pros and Cons of Paper and Digital Diaries

Type

Best for

Drawbacks

Paper Diaries

Smaller budgets

Creative assignments

Low-tech respondents

Easy access to diaries during follow-up research

No research access during data collection

Hassle factor – printing, shipping, collecting, scanning, etc.

Digital Diaries

Capturing data at the time it is happening

Engaging clients

Learning about respondents before they get in the room

Pre-screening respondents for follow-up research

Can feel sterile – need to work to create a warm environment

May not be a fit for all respondent types

Cost not usually free

Diary Exercises to Try

Ready to take your diary exercises to the next level? Below are a series of six activities to try that go beyond asking respondents to jot down every time they do X or purchase Y. All of the activities provide a basic structure, but they still leave room for you to customize and improve them to meet the needs of your next study.

What about platform?

A number of these exercises are best executed on a digital platform. The platform providers change all the time; for an up-to-the-minute guide, I would recommend checking out the New Qualitative Guide published by the Qualitative Research Consultants Association (QRCA) or, better yet, talking to QRCA colleagues who work with digital platforms regularly.

Although many of these exercises can be accomplished using free platforms, tools developed specifically for the marketing research industry make it really easy for moderator, client and respondent to participate in ways that are seamless and appropriate. Sometimes spending a few extra bucks on a paid platform is worth the investment in terms of your time and sanity.

Exercises

Timeline

Often, a diary seeks to capture a particular behavior over a time period that may vary by respondent. For example, a consumer in the market for a new car may research, shop and test drive for weeks before making a choice, or he may walk onto the lot and make a purchase rather quickly. The Timeline exercise is simple. Give each respondent a large piece of paper with an arrow drawn across it to represent time. Each time (s)he does something related to the activity of interest, (s)he notes it on the timeline. In this way, behaviors are captured as they happen but don’t need to follow a specified format.

Photo Album

We know consumers are already taking pictures of every aspect of their daily lives. Why not harness that for marketing re- search purposes? A photo-album diary exercise goes beyond having respondents take photos by asking them to organize those photos into a meaningful story with rich captions. And when the study is over, you can cherry-pick the best albums and use one of the many online tools to actually create hard-copy albums that the client team can use and refer to for months and years to come.

Moments

When we want to know about the emo- tions surrounding a particular behavior, the “moments” diary exercise is ideal. Consider the difference between “Take a photo of your morning coffee” and “Keep a journal of your coffee moments.” Ask respondents to tell you as much as they can about the moment — what was going on, how they felt and what made it a moment.

Easy & Hard

Often, clients want to capture product usage so that they can figure out what is working well and what is not. In these cases, consider structuring your diary exercise to capture what is easy and what is hard about using a particular product or service. In fact, this exercise works well with any polarity: How about yum/yuck, best/worst or fun/boring?

“Love is…”

Sometimes, it is important to understand how consumers think about a big attrib- ute such as love, clean or healthy. But getting them to articulate what these things mean can be tough. (How do you define clean?) A diary that asks respondents to capture photos that encapsulate a particular attribute can quickly get to higher-order benefits and provide agency or creative teams with a ton of fodder for creative development.

Deprivation Day

Deprivation day is not an exercise unto itself, but rather something to add on to a longer diary assignment. The way it works is this: If you are having respond- ents log some activity or product usage for a defined time period, ask them to spend a day going without that activity or product. For example, if respond- ents spend a week capturing “coffee moments,” ask them to go without coffee for a day and journal what hap- pens. What did they eat or drink or do instead? How did it feel? (You might call these “Non-coffee moments.”) A word of warning: Tell respondents dur- ing the recruit that you are going to ask them to do this; some might not be able to go without.

Final Words of Wisdom

Whether you are doing your 1st diary study or your 100th, you are never alone in the process. The technology providers all have relevant guides and case studies to spark your imagination, and I have found that many of them are willing to give you a free trial to get you started learning their platforms. After all, your success is their success! So, why not add a diary activity to your next project, and see what you can learn?

Conducting Field Studies with Older Adults Lessons for Recruiting and Testing Older Users

March 15, 2009 No Comments

Last winter, we ran a usability study in our lab with adults aged 75 and above. This age group is often classified as the “older-old” population, as opposed to the “younger-old” population of 60-75. We had done considerable research with this older-old population before, and we knew to expect higher-than-average cancellation rates because of participants’ health and transportation issues. But when a light snow wreaked havoc with our testing schedule, we decided to take the research to our participants. In doing so, we discovered that field testing and field research work especially well with older adults.

We’ve conducting over 100 interviews and field tests with adults 75 and older in the past two years, mainly in the area of consumer products and home healthcare devices.

Recruiting Older Participants

One of our first challenges, as in any study, was finding the right participants. For us, this meant people aged 75 and older with a variety of health-related issues who would allow us to conduct interviews in their homes. In recruiting this population, the following strategies worked well for us:

Establish relationships with directors of senior housing facilities

We found that senior housing facilities and assisted living facilities gave us great access to potential participants. Focusing on these facilities also cut down on the travel time between interviews. When we explained our study to the directors at the centers, they often became great advocates for our research. They circulated information for us and, in some cases, even arranged for us to speak to their residents to explain our studies. This allowed us to clarify the purpose of the research and reassure residents that we were not selling anything. Having someone they trusted vouch for us helped our participants feel more comfortable with our studies.

Reach out to those who care for older adults

Paper flyers and internet postings that targeted caregivers, as opposed to older adults themselves, proved very effective. Again, this provided older adults with the reassurance that someone they trust-often a son or daughter-had learned about the study and found it legitimate. When working through caregivers, however, it is important to ensure that the actual participants themselves have agreed to participate and are not being pressured by their caregivers.

Use your social and professional network

Networking was an effective way to locate older adult participants. For example, one of us had a late cancellation for a study that involved an international client who was leaving the next day. At the last minute, she was able to replace the participant with the mother of the contractor who was working on her house!

Our early round of interviews also provided excellent networking opportunities. Our first participants told their friends about the study, who then wanted to participate in later rounds of testing. However, this method has potential pitfalls. For example, we encountered a group in one senior housing center who took down posted flyers because they wanted to reserve participation for their circle of friends.

Interacting with Older Adults

Maintaining clear and open communication while we interacted with older-old participants contributed to the success of our studies. Being trustworthy, patient, and punctual also enhanced our interactions during the sessions.

Support the effects of aging in written communication

To support the visual decrements of aging, we increased the font size in our flyers and used bold text. We also needed to support the effects of aging on cognitive functions such as memory. We made flyers as straightforward as possible, with simple sentence structure and no unnecessary details. In a diary study, we provided a list of clearly stated reminders, labeled, “Important things to remember,” for participants to reference on their own. Consistency also proved to be important. In an early study, our informed consent form had the line for the signature first and the line for the printed name second, while our cash receipt form had the order reversed. When we presented the second form, some of the oldest participants signed the Name line and then were embarrassed when they read “Signature” on the next line. In subsequent studies, we fixed the forms to present the name and signature in a consistent order.

two different forms — Figure 1. Inconsistency between signature lines on forms can cause problems.

Support the effects of aging in verbal communication

We were able to support hearing impairments by speaking loudly and clearly, and in some cases, participants asked us to sit on the side nearest their “good ear.”

As with written communication, we strove to be clear and concise and to avoid long, detailed instructions. Asking older adults to retain many details at once taxes their working memory, which becomes especially vulnerable with age. We chunked the information, stated each point clearly, and left time for questions between topics.

Gain their trust

Older adults are often targets for scams, so it was important for us to be open about our research and to answer all questions, even if we had to repeat ourselves. During a recruiting presentation at one assisted living facility, we were careful to state that we were not selling anything and that we would pay honoraria for participation. Still, in the end, some residents asked how much it would cost for them to participate in our study.

We limited our team to one or two interviewers, so that we would not overwhelm participants, and made sure to tell them ahead of time how many of us to expect.

We also invited caregivers to be present for interviews, both for their peace of mind and for participants’ reassurance. One caveat, however, is that sometimes caregivers, who feel very connected to their loved ones and want to help, may answer for the participant. In these cases, we gently redirected the question, saying, for example, “Now, Mary, what do you think?” We were careful to ask for participants’ permission before recording or taking photographs. We asked them during the screening call and again at the beginning of the interview. We used portable video cameras to record usability tests. For field interviews, we took digital photographs and handwritten notes.

Be on time

Arriving on time is very important in establishing participants’ trust. Our participants were often anxious for us to arrive at their homes. To ensure on time arrival, we allowed extra travel time to account for unforeseen delays, such as traffic. Participants often expressed appreciation for our punctuality and found this gesture considerate and courteous. It set a positive tone for our sessions.

interviewing with video camera — Figure 2. Equipment for recording usability tests in the participants home.

Be patient

As busy researchers, we were aware of all that we wanted to cover in our interviews. But our participants treated us like company in their homes and, as guests, it was important to allow time for them to tell us about their families, and not to jump right into the interview questions.

In answering questions, participants sometimes needed to give us more detail than we really needed. For example, we might ask how long they lived in their current residence, and they would answer with a story about when and how their spouse died, how their children worried about them remaining in their family home and encouraged them to move into the senior housing facility where they currently lived. Paying attention to these stories not only showed respect for our hosts, it gave us valuable insight into our participants’ lives, including their use of our clients’ products.

To account for this deep level of communication, we allowed about 50 percent more time for these interviews than we would with a younger population.

Be prepared to probe

Probing is a standard interviewing technique, but with older adults, we sometimes found it necessary to probe around even simple statements. For example when we asked, “How is your general health?”the majority said, “Good.” However, further probing indicated some had had cancer, heart surgery, respiratory disease, or other ailments. For these participants, ìgoodî seemed to mean “Good for someone my age,” or “Good, considering all I’ve been through.”

In probing, we were careful not to be intrusive, which can be a delicate balance. While we wanted to know of any health concerns that affected our research, we did not want to elicit medical information that was personal and/or confidential. When one participant offered to show us all his medications, we politely declined.

schedules on a refrigerator — Figure 3. One participant’s schedule of activities on the refrigerator.

Older Participants’ Involvement in Research

As we interviewed older participants, we learned more about how they related to the research in general and how they understood their participation in our studies.

Older participants were highly committed to the research

Our participants took their involvement in our studies very seriously. Only one of sixty participants in our diary studies dropped out-and she did so because she was hospitalized. Participants were also highly motivated to complete their diaries accurately. To help aid their memory, some carried note pads and recorded activities as they occurred so that they would not forget. We have conducted similar diary studies with younger participants and did not find nearly this level of commitment.

Participation benefited the older participants beyond compensation

A number of participants told us that being involved in the research was a positive experience for them because it made them feel needed and valued. Several said they were pleased to be able to do something that would help others in the future. The research also appeared to stimulate some older participants because it encouraged active engagement in a task. One of the facility directors reported that she felt that our diary study actually “exercised” the memories of residents at her facility.

Compensation was still important

As with other studies, we paid our older participants at the end of each interview. While a number of participants told us they would have been happy to participate “for free,” they clearly appreciated the honoraria we gave them. Compensation not only supplemented their income, but it helped express the value we placed on their time.

Older participants are each unique and vary in many different ways There is huge variation among persons of any age in personality, as well as cognitive and physical functionality. We saw participants of the same age who could hardly walk and others who exercised for an hour or two per day. While some participants had little activity beyond watching television, we took pictures of one 93-year-old’s calendar and refrigerator door, which indicated a life full of activities and responsibilities. We met an 88-year-old who still tutors teens in a city school.

We were amazed at how frank and open our participants were in our interviews. They sometimes told us things they wouldn’t tell their children–that they took naps in the middle of the day, for example. One woman told us that she was looking forward to going to her next home. We asked where she was moving, and she responded “Heaven…I hope.”

Final Thoughts

Our fieldwork with older-old adults opened our eyes to their needs and their strengths. By being trustworthy and adapting our communication to meet their needs, we were able to recruit participants and conduct successful field studies.

As with any study, we became sensitive to the issues facing our target population. With our older participants, we found ourselves looking ahead at our own future and wondering how we will fare. Thankfully, our work in this area has the potential to positively influence the lives and health of current and future senior citizens.

Cultural Probes: Understanding Users in Context

September 15, 2006 No Comments

One of the latest techniques in the Human Computer Interaction (HCI) community is a data collection mechanism called cultural probes. These were introduced in 1999 by William Gaver, an academic based at the London Royal College of Art, in the publication Interactions.

Since then, cultural probes have been discussed, developed, and used extensively in academic HCI research. Although industry practitioners are starting to adopt cultural probes, there are still many questions around the usefulness and practical application of this technique. This is not a simple question and, as Gaver says, “The use of probes is geared toward design generation, not problem solution.”

What is a Probe?

So what exactly is a cultural probe and how would you explain it to your skeptical boss in the elevator? Gave describes cultural probes as “designed to prompt and elicit information from people about their lives and local culture.” A simple explanation is that it is a diary maintained by a user over several days or weeks. Typically, the diary includes photographs taken by the user, pictures cut from magazines, voice recordings, and written notes. The technique is also called “diary study,” “media diary,” and “photo collage.”

To create their diaries, users are given a “pack.” The contents depend upon the objectives of the data collection, the characteristics of the users, and the imagination of the designer, but common items are scrapbooks, pens, cameras, and diaries.

Some of the more innovative and interesting items that have been used in packs are catch phrases printed on sticky labels and simple digital memo-takers that are repackaged as dream recorders. These let users record ideas or potential requirements as they occur to them, such as when awakening from a vivid dream.

Users are briefed on how to use the packs and what information to record. They are asked to create the diary independently, in their own environments, with regular entries over an allocated time. The designer may contact or visit the users occasionally to monitor progress, answer questions, and keep the users focused.

photograph of notebooks with pictures and post-it notes attached — A typical cultural probe

Why would you use Probes?

All data collection techniques have advantages and disadvantages. To understand why probes might be useful, we need to consider the reliability of data constraints on time and money and the need to collect data for non-work or non-office activities—for example, the use of mobile devices for social, leisure, or on-the-road usage.

Reliability of data is a fundamental challenge for all designers. Every data collection technique risks unreliable data. For example, focus groups may drift so far from reality that the descriptions of actions bear little relation to actual behavior. In laboratory usability tests, the lack of real context may distort behavior. In contextual enquiry, it may simply be impossible to control observational conditions due to the noise of other activities unrelated to the project at hand.

Cultural probes are an additional tool with a new set of strengths and weaknesses. For example, the reliability of data may be enhanced by users being experts in their own lives. Cultural probes provide the opportunity for users to document their lives in their own context, with minimal interference from the designer. Probes also provide the opportunity to triangulate against another set of independent data (such as focus group, usability tests, or survey findings) and thus improve reliability.

Time and money constraints may be circumvented. Commercial organizations often expect HCI practitioners to somehow have an inbuilt and expert understanding of the target audience, and they may make little allowance for the time-consuming and expensive process of collecting and analyzing data to gain a significant understanding of the audience. Because cultural probes place the onus of the actual data collection on the user, practitioners can focus their effort on the analysis of data.

Access to interaction and behavior for non-work activities, or work performed outside the traditional workplace, is a new challenge for commercial HCI. The traditional work environment tends to be well-defined and accessible. Work within this environment is generally routine and structured. People are often able to describe their work, working environment, and patterns.

However, the contexts of both non-work and leisure time activities are much harder to define, and any data collection technique requiring the designer to be physically located with a user may disrupt or destroy the quality of the activity. Furthermore, the lack of routine and structure in these activities may greatly expand, or make uncertain, the amount of data collection required. This is one of the most significant advantages of cultural probes. Data can be collected easily despite the unstructured, non-routine, and easily disturbed context of the activity. Users frequently become very engaged in the studies, and often comment that keeping diaries and scrapbooks is fun.

Challenges using Probes

As with all data collection techniques, probes have challenges. The most significant of these is ensuring that users understand and maintain focus throughout the data collection process. This is not trivial, as probes are very much a hands-off data collection technique; the user is neither observed nor directly guided during the day-to-day creation of data.

On the other hand, the designer must be careful not to be too restrictive or prescriptive and thereby lose ideas, inspiration for design, and unexpected information about user behavior and interactions. The risk is that a lack of focus may lead to very noisy data that is difficult to analyze and does not provide useful input to a specific design. Probe studies should include an initial briefing session and at least one interview visit or telephone call to the user during the early part of the diary-keeping process. This is extremely valuable as many probe users need to be focused, re-focused, or simply taught how to use the pack materials.

Despite the reduction of time and effort required from researchers (since users collect their own data), probe studies consume elapsed time, which may be in short supply in tight project schedules. Therefore, probe studies should be run as early as possible in projects.

Researchers must also take care to gather an appropriate amount of data. The number of users, the length of the data collection period, the probe pack materials, and the requested frequency of diary entry must all be taken into account in the design of cultural probes. For studies in which I have been involved, about twenty to thirty minutes per day was asked of users. This may vary, with some days where less or no data is entered into the diary, and other days where more data is entered.

The length of the data collection period is also important. For example, one study involved seven weeks of data collection but five weeks would have been enough. Some of the users lost interest, finding that some of the data contained nothing new and there was simply too much data. At the other end of the scale, in an industry study for a specific product, the challenge was determining the minimum amount of time to make the study worthwhile. We tried seven days and found that to be just enough. This was mainly due to the simple and easily definable area of focus for the study.

Another influence on the quality of data is the level of interest and participation of the users, as they are not only the source, but also the transcribers of the data. Users may volunteer, ideally with the interest and personal motivation to support diary filling over an extended period. Alternatively, if you need to closely match a target audience persona, or have practical concerns such as time constraints, you may need to recruit and pay users. This is usually done through agencies. In this case, you must take care to maintain the users’ interest, and remuneration must be carefully considered.

Examining and Using Probe Data

Probe data is not straightforward to use. In the first place, probes can generate tremendous amounts of data, requiring significant time even for a first reading. The more complex question, however, concerns how to examine and analyze the data. Academic researchers have spent considerable effort trying to understand how to interpret the data from probes. Gaver comments in one study that, “Our probe results are impossible to analyze or even interpret clearly because they reflect too many layers of influence and constraint.” However, in another part of the same study he comments that, “Sometimes the trajectory from probes to designs is relatively straightforward, and design ideas can clearly be traced back to probe returns.”

There is currently no one standard method for using the data from probe studies to influence design. The most straightforward method is for the HCI practitioner to brainstorm design ideas. The data can provide a rich insight into the lives of users.

Probe data also generally provides a good understanding of a real-world domain. The data enriches the ideas of the designer or researcher and provides a supplementary influence rather than a direct prescription for design solutions. In particular, probe study data, together with interviews and debriefing, provides an excellent source of data from which personas can be constructed.

In one project, probe data was used to create scenarios which, in turn, were used as the basis for participatory design sessions. The data informed and enriched the scenarios, providing excellent context and motivations. The participatory design personnel included researchers involved in the study, HCI experts, and the probe users. It was found that the probe users, while having an intimate knowledge of their own data, did not have the HCI skills for designing, and that the best design concepts came from HCI experts who were familiar with both the probe data and the context of the project as a whole.

Probes for Industry

Cultural probes are a new technique worthy of attention from commercial practitioners. They are particularly useful for novel non-work design situations where user behavior is relatively unknown or difficult to access. Probe data can be very noisy, however, and the data at times may be incomplete, unclear, and biased. It is important to set expectations that the data will enrich the understanding of the target audience and support the creation of personas and scenarios; but while it is intended to support design solutions, it generally does not explicitly contain a design solution.

Perhaps expectations for the use of probe data are best summarized by Gaver, who stated, “Most of the time the relationships between probes and proposals are more complex and difficult to trace. Our design ideas are formed from a combination of conceptual interests, technological possibilities, imaginary scenarios and ideas for how to implement them. The probes are one influence in all this.”

Probes are perhaps best implemented in industry as a technique to supplement standard industry design processes such as initial analysis, feasibility, and early design work. To ensure that the focus of the project is maintained and the effort is contained, designers should take care to plan the materials in the probe pack, set the data collection time, and predetermine the type of data examination or analysis. Probes force the designer to think about the context and motivations of use, thus building a stronger understanding of the target audience and potentially leading to better and more informed designs.

Chronicling Pain: Creating Dynamic, Customizable, Modular Icons for Describing Abstract Physical Symptoms

December 19, 2023 No Comments

Background

In terms of healthcare, a broken leg is relatively straightforward to manage. It is painful, inconvenient, and requires healing time, but most communities readily offer a seat on the bus, employers are likely to accommodate a request to work from home, and doctors can quickly diagnose and treat the injury.

The same cannot be said for abstract, invisible illnesses, such as chronic pain. Even with a diagnosis, it is challenging for patients to convey their need for accommodations if they appear healthy. They don’t necessarily rely on mobility aids like canes or wheelchairs. For instance, convincing another person to yield their seat on public transport during a fibromyalgia flare-up may be difficult. Moreover, obtaining a diagnosis for a condition of chronic pain can prove challenging.

Chronic pain can often strike without warning, apparent physical injury, or traumatic event. It can be sudden, unexplained discomfort. I experienced this firsthand at the age of 14 when I woke up one day with excruciating pain on the left side of my body from the top of the shoulder to the foot. My symptoms included sharp pain, tingling, burning, numbness, and a sensation of fluid in my leg. These symptoms persisted intensely for months, with occasional flare-ups still occurring over a decade later.

The most significant challenge I faced was that these sensations were so alien to me that I could not describe them; because I was experiencing discomfort, I kept telling my primary care physician that I was in “pain,” but the sensations were so much more complex than that. There were times when, for example, it was more that my leg felt like Jell-O®, and the absence of feeling or pain was what was unbearable.

During these appointments, I was asked to use conventional pain rating icon sets such as the Wong-Baker scale to describe pain intensity. Emoticons range from a smiling face to a frowning face (Figure 1). I found that the use of simplified emoticons was not a meaningful way to measure my pain as my responses would often be affected by other factors such as my mood. This was largely because I got used to the pain over time and just started feeling a general sense of malaise, even though the pain was not actually improving.

Other icon sets aided in the description of different pain sensations. For instance, a black-and-white drawing of flames would represent a burning sensation (Figure 2). I found that these illustrative icons would depict their associated adjective but not capture the actual sensation, minimizing their impact in clinical conversations I was having.

Figure 1. An example of a conventional pain scale using faces. (Credit: Lord Belbury via Wikimedia Creative Commons)

Figure 2. An example of a highly simplified burning pain icon. (Credit: Pike Picture via Noun Project Creative Commons)

The drawbacks of conventional pain scales are also discussed in Melissa Rach’s article for the Mayo Clinic, Pain Scales: Valuable Tools or Problematic Practice? She notes that they fail to account for meaningful variables such as pain tolerance, pain history, emotional state, pain changes with activity, and pain fluctuations over time. She also explains that numerical scales in particular can create “unrealistic expectations” and thus disappoint many patients by implying that their pain could one day reach a zero on the scale, which might not be the case. Rach also found that pain scales encourage the overuse of pain medication to mask pain and lower the reported number on the scale instead of treating or managing the underlying problem long-term.

When attempting to elaborate on my responses to these pain scales and icons, I had vivid visuals in my head of what I was experiencing, but I could not put them into words. Whenever I tried, my doctors would repeat back to me things like, “Ok, so you have a bit of a dull muscular ache,” which was the complete opposite of what I thought I described. The advice I received could seem very dismissive. “Try stretching a little or flip your mattress over; sometimes those can relieve aches…” did not account for symptoms such as the periodic loss of sensation and use of my left arm or my toes spontaneously twitching. Furthermore, without a formal diagnosis, I encountered skepticism from teachers and peers at school as I was frequently absent due to being bedridden by my symptoms and could not provide a definitive answer to “What’s wrong?” or “What illness do you have?”

To this day, I have never received a formal diagnosis or treatment plan, and I have ended up just managing my pain through medication, rest, and sports massages when experiencing flare-ups.

User Research

For my final degree project at the Rhode Island School of Design, I explored how color, shapes, motion, and sound could enhance the effectiveness of icons that represent pain. I hoped this could improve communication between patients with chronic pain and their healthcare providers (HCPs), family and friends, and the general public.

Pain Diary

To begin with, I maintained a visual pain diary over four weeks (Figure 3). A pain diary is typically a written record that tracks occurrences of pain, how bad the pain is, and whether treatment is helping. Diaries are often used in the initial stages of chronic pain diagnosis. For this exercise, I dedicated approximately 60 seconds a day to sketching any pain I experienced on Adobe® Illustrator®. This time constraint aimed to capture my intuitive use of colors, shapes, and gestures.

Figure 3. An example of my pain diary using graphic illustrations.

Once I completed the pain diary, I distilled the various gestures into icons representing ten common types of pain: burning, cramps, dull ache, freezing, migraine, neuralgia, numbness, stabbing, throbbing, and tingling. I then integrated sound effects and motion and distributed the icons to 20 participants with chronic pain via a digital questionnaire for user research purposes.

The questionnaire also covered participants’ general experiences with chronic pain and symptoms, their experiences with diagnosis and clinicians more broadly, their current treatment plans and activities they would be willing to try, and colors and sounds that relieved their pain.

Research Methods

My study was self-designed under the supervision of my thesis advisor, Nancy Skolos. I opted for a sample size of 20 individuals experiencing chronic pain due to the time constraints of my initial thesis project, which was limited to approximately 12 weeks. My goal was to gather a variety of responses while maintaining the depth and detail necessary to draw meaningful conclusions. After some consideration, I determined that a participant pool ranging from 15–30 individuals would provide a diverse, yet manageable dataset.

To recruit participants, I tapped into multiple sources, including mutual acquaintances and chronic pain support groups. I ensured that participants exhibited a broad spectrum of prominent symptoms and diagnoses, encompassing conditions such as injuries, multiple sclerosis, fibromyalgia, and migraines.

For the data collection method, I settled on a one-time digital questionnaire for several reasons. Time constraints played a significant role, but it was also during the peak of the COVID-19 pandemic. Given that individuals with chronic pain are more likely to be immunocompromised and unlikely to participate in in-person interviews, a digital approach proved both practical and safe. The anonymity of this method was also crucial as it encouraged candid and comprehensive responses, particularly regarding the origin and background of participants’ pain. Anonymity was maintained by not asking for names, photos, or any other personally defining information. Participants were also asked to check a box indicating whether or not they would be happy for the details of their condition and treatment to be used anonymously within any part of the project, now or in the future.

As this project progresses and undergoes further refinement and testing, I do intend to expand my research. I plan to work with larger groups, ideally through in-person or video interviews conducted over extended periods in order to obtain more comprehensive results.

Findings

The participants’ first task was to assess the accuracy of my drafted icons and propose edits if needed. To reduce participant effort, especially considering potentially limited energy and physical discomfort, I provided visuals for their responses rather than requiring them to create their own. The results reinforced my thesis in several ways. Accuracy ratings were nearly evenly split, and the suggested adjustments from those who found the icons inaccurate varied widely. For example, where some participants recommended softer, warm-toned imagery that moved gently from side to side, others preferred sharper, cool-toned icons that pulsated. These findings strongly indicated that words alone were insufficient for describing pain, as the same term could have multiple interpretations and lead to misdiagnosis.

Many studies support these findings. For example, a study by Salovey P., Seiber W.J., Smith A.F., et al., published by the National Center for Health Statistics in 1992, found a significant disparity between the types of pain that patients associated with the same adjective, averaging ten types among those with chronic pain and 18 among those without. Furthermore, a 2016 study by Sharma S., Pathak A., and Jensen M.P. published in the Journal of Pain Research, found that individuals in Nepal typically use sensory metaphors and similes to describe their pain rather than single terms (for example, “feels like an ant bite”) and do not have definitive terms for concepts such as achy pain.

As stated above, this study was limited by factors such as time and therefore sample size and variety of methods used, so I acknowledge that more comprehensive data will need to be collected from additional research to refine and optimize the project. However, following the synthesis of my user research findings, I moved forward with designing a sample set of multimedia icons as a minimum viable product offering a fundamental solution for the lack of clarity in verbal descriptions demonstrated during the study. I also recognized the need for added customization, including the icons’ names, visuals, and sounds to accommodate further nuance and specificity.

In addition to a refined icon set, I decided to design a proof-of-concept application to demonstrate how the icons could be customized and used to log symptoms daily. The app would also include educational, pain management, and doctor-patient communication features inspired by the feedback received in the second part of the questionnaire.

Naming and Branding

For project naming, I settled on sentoms™, a portmanteau of sensory and symptoms. The logo incorporates the Work Sans font, chosen for its web-safe readability, which is particularly important when users are in pain and need quick access to information. This font also adds character without being overly complex. The color scheme for the logo and brand stems from feedback received during user research that participants find these colors calming and pain-distracting. The brand identity aims for a friendly and soothing aesthetic to avoid clinical or pain-evoking associations (Figure 4).

Figure 4. sentoms branding overview.

sentoms Icons

Figure 5 shows stills of the refined sample set of sentoms icons, crafted based on some of the responses from the user research questionnaire. An online video shows the icons with full motion and sound.

Figure 5. sentoms icon stills.

Despite recognizing the need for customization, I still chose to create standard icons that could be heavily modified rather than making users create icons from scratch. This was to make the process less overwhelming and arduous, especially as users would be in discomfort when using the icons to log symptoms and may also have limited dexterity, patience, or stamina. Ongoing work focuses on refining and eventually converting these into fully editable modules, utilizing a creative coding library such as p5.js or a vector animation code format such as Lottie.

I also created static versions of the icons, designed for situations in which motion is impractical, such as print materials or mapping symptoms on a 3D model of a patient’s body (Figure 6).

Figure 6. Static sentoms icon alternatives.

sentoms CP

I subsequently designed sentoms CP to serve as the functional proof of concept for the sentoms icons.

Overview

sentoms CP is a mobile application enabling patients to create and tailor sentoms, log their pain symptoms using the icons, and view the icons mapped onto a 3D model. Because the app allows users to change the visual intensity of sentoms through color, shape, motion, and sound, it is easy to see progress and relief at a glance as the icons are edited and logged over time, which enables meaningful clinical conversations to take place while motivating users to continue with their prescribed treatment plans.

The app also provides users access to additional resources, including general information about chronic pain (curated by HCPs), chronic pain case studies, direct messaging with HCPs, and personalized physical and therapeutic exercise plans designed by HCPs to alleviate pain symptoms.

Users have two options for exercises: the To-Do List and the Exercise Bank. The To-Do List is the set of exercises curated by the user’s HCP, and daily reminders can also be set. The Exercise Bank is the full range of exercises available, broken down into which symptoms/conditions they are recommended for. This gives users the autonomy to complete additional exercises throughout the day if they choose. Additionally, exercises that are comparable to the exercises on the To-Do List are marked as Fulfills To-Do List, meaning that users can do these exercises instead of the prescribed ones if they find them more comfortable or enjoyable. The original exercises on the To-Do List will still be checked off.

Examples of prescribed physical exercises include yoga, Pilates, and walking in place, while therapeutic exercises include meditation, guided imagery, sound therapy, and cognitive behavioral therapy. I included exercises in both categories based on responses given during the user research phase and additional literature that discusses the roles of physical therapy, meditation, and virtual reality in pain relief. While doctors focus on the physical aspect, many patients also experience associated feelings of hopelessness, depression, anxiety, and a general lack of motivation, which can be overlooked. Providing activities that promote mental health might have direct and indirect benefits for patients who will then feel more comfortable sharing and logging aspects of their experience with chronic pain.

Conversational experiences utilizing AI chatbots are a popular healthcare solution for providing immediate feedback. However, I opted to enable direct messaging with the user’s real HCP in this scenario because the core value of the sentoms concept is optimizing accurate communication about pain between humans.

Design Process and Final Prototypes

The design process began with wireframing to facilitate an understanding of how various application components would interact (Figure 7).

Figure 7. Mobile app wireframes.

Subsequently, multiple rounds of visual design refinement took place. Initial attempts aimed for a cheerful, positive aesthetic to avoid triggering reminders of pain. However, feedback led to simplifying decorative elements and allowing users to choose a single color for the entire application instead of using extensive color coding. Figure 8 shows the evolution of the interface.

Figure 8. Mobile app visual design iterations.

Upon finalizing the visual aspects, the prototype was developed. Figure 9 demonstrates some key screens from the video walkthrough.

Figure 9. Mobile app screens.

Moreover, the mobile app can seamlessly integrate with a wearable/smartwatch app to enhance convenience in symptom tracking and exercise monitoring. This addition was prompted by user research responses emphasizing the motivation they gained from tracking progress and achieving specific goals. Allowing users to witness improved abilities in prescribed activities, as well as a general increase in physical activity, over time boosts morale. Because HCPs can monitor app data, patients can also be held accountable for their adherence to treatment plans, and these plans can be adjusted to be more accurate when necessary. Figure 10 shows some key screens from the video walkthrough.

Figure 10. Wearable app screens.

Future of the Project and Broader Applications

I am currently dedicated to refining and perfecting this project while looking to involve larger groups of individuals living with chronic pain for further research and testing.

As it stands, this project sheds light on the limitations of existing pain description systems in healthcare. It also demonstrates the potential of dynamic, customizable, modular icons to enhance communication about pain. My firsthand experiences and the diverse responses from user research emphasize the importance of customization, nuance, and specificity when describing abstract and often invisible symptoms to healthcare professionals.

This concept is invaluable for the attending healthcare professionals and other clinicians collaborating with a patient, such as third-party consultants, telehealth practitioners, rehabilitation facility staff, home care providers, or family caregivers. Maintaining a comprehensive digital record of a patient’s pain symptoms and their real-time adherence to prescribed exercises is invaluable for gaining insights into concurrent symptoms, facilitating diagnoses, and determining the appropriateness of additional treatments. Furthermore, handwritten notes or hand-drawn sketches can often be illegible, making digitized communication a crucial asset for future reference.

The sentoms multimedia icons also have the potential to be used in campaigns that educate the general public about chronic pain to provide insight into what individuals with chronic pain may be experiencing, even when their condition is not immediately obvious.

Beyond chronic pain, this concept embodies an inclusive and universal design proposal. It can empower individuals with cultural, social, language, and class barriers, as well as groups like children and those with non-verbal autism, to effectively describe abstract symptoms they may be encountering in a clinical setting.

What Shall We Watch Tonight? Using Mobile Devices to Plan TV Viewing

September 15, 2007 No Comments

Interactive Television (iTV) offers an exciting future full of dynamic viewer-channel interactions. However, while increasing numbers of consumers purchase iTV capable devices ,and channel providers experiment with services, a range of usability challenges remain.

One problem that warrants investigation is that of planning what to watch on television. With an increasing number of available channels, each offering many different programs, it is clear that many people may develop a major headache navigating through the maze of possibilities to find a program they want to watch.

Electronic program guides (EPGs) developed to help television viewers deal with this increasing number of channels are now regularly used by many people. Program guides display the television schedules on the television screen and allow viewers to choose what to watch directly from the on-screen list.

Although they are certainly popular, program guides on their own are not necessarily best suited to the task of television planning. Unlike printed television guides, they can only be used while in front of the television set—the user cannot use them while on the way to work or waiting for a train to arrive. Furthermore, the viewer must be focused on using the system and be actively engaged in setting viewing preferences and browsing the on-screen listings. This does not come naturally to most television viewers because viewers like to “lean back” and passively watch television, rather than “sit forward” to interact with their television set.

Our research efforts have been focused on how handheld technology—mobile phones, Personal Digital Assistants (PDAs), and the like—present an opportunity to augment traditional program guides and to better support television planning. Because it empowers viewers to plan their television viewing anywhere and at any time, it means that their natural style of interaction is better supported.

a list of tv shows and what channels they play on — Figure 1. An Electronic Program Guide (EPG)

Electronic Program Guides

A wide range of research has been conducted to look at how best to design program guides. These include studies evaluating the usability of program guides and studies looking at how peoples’ television viewing and planning habits might influence and inform the design of future program guides. Such studies often result in design guidelines and recommendations. However, existing guidelines are primarily directed toward on-screen guides, accessed directly via the television set. No such guidelines exist for a program guide running on a handheld device, or for a system integrating an on-screen program guide with a handheld device.

Some program guides have made the transition to handheld devices, but all are very much technology driven. The push appears to have come from the technology behind the personalization and artificial intelligence features, and not necessarily from any identifiable user needs for such a system. Furthermore, existing handheld systems seem lacking in any real integration with their homebound TV counterparts. They provide recommendations but do not allow users to directly act upon them, for example, by requesting recommendations to be recorded. The only notable exception to this is PTVPlus-GuideRemote. This system not only enables a user to directly control the television, but also provides personalized television listing information and program recommendations. However, being based around a universal remote control and not a PDA or mobile phone, the system clearly has limited scope for use away from the television set.

Multiple Device User Interfaces

Multiple device user interface systems for digital television tend to be oriented toward either delivering what has been termed “extended television”—that is, giving people further opportunities to view or participate in a program away from their TV set—or toward directly controlling the television channels and functions using a handheld device. However, not much work has been directed toward television planning.

Extended television can be seen as an attempt to extend digital television beyond the television set itself, allowing the extension of television brands to a range of different electronic devices. So, viewers can feel connected to a television program through a mobile phone or the Web even when it is not on air and there is no television within reach. Such services can involve synchronous interaction, where the interaction occurs while the television is turned on, or asynchronous interaction, where the interaction occurs either before or after the program has been broadcast.

Paces of Interaction

As users’ context changes, so might the desired level of interaction with the technological resources available to them. We might then differentiate between the “lean-back” interaction seen in conventional TV viewing, and the “sit-forward,” highly engaged form seen with computer use.

Studies all agree that television traditionally supports a very passive style of interaction, but that the level of interaction increases with the introduction of increasingly interactive services, such as program guides. There is a clear conflict between the traditional style of television interaction and the more interactive television services. It is therefore not surprising that studies have shown that not all viewers are eager to switch to a more involved level of interaction, preferring the old and the reliable to the new and suspect. Researchers have sought to reduce this level of interaction by pushing more personalized content through program guides. However, to date, little work has investigated the possibility of moving it away from the television and onto a different platform, such as a PDA or mobile phone.

Television Planning Diary Study

We asked eight people in the United Kingdom, between ages 18-35, to keep a week-long diary of their TV planning activities. We asked them to note the resources they used, when they were used, and for what purposes. We then used these entries as a basis for a mid-study phone interview and an end-of-study face-to-face discussion.

Figure 2. Example of a diary entry.

Planning Can be a Social Activity

All diarists indicated that they often watch television with other people. This seems to have a real impact on how they plan what to watch because consideration must be given as to what others want to watch.

Although most diary entries were of individuals using television planning aids, a number referred to collectively planning when seeking an agreement about what to watch together. Changing channels or on-screen television planning aids are used to enable everyone to see what programs are available. Perhaps the ingrained social nature of watching television makes on-screen television planning aids more suitable for collective planning. Watching television is frequently a social activity, so it seems natural that using a television to plan television viewing also would be a social activity.

Television can also provide a social commonality. For example, diarists spoke of chatting to friends about programs that they watched. There was also evidence of diarists recommending programs and movies to friends. One diarist spoke of a friend with whom he regularly sent and received text messages with television program recommendations. Another saw a program that she felt a friend might be interested in and phoned to let them know that it would be on. There appears to be evidence of people not just looking for themselves, but also looking for television programs that might interest others.

Viewers Look for More of What they Like

The type of program and whether the diarist had seen it before and liked it were the most important factors when deciding what to watch. The channel broadcasting the program was seen as being important because it is recognized that some channels—such as MTV—only show programs of a certain type (music in this case). Furthermore, there is a clear issue of trust. It is recognized that some channels, primarily the terrestrial ones in the UK, can be relied upon to show programs worth watching. That is, the quality of the content is seen as being higher than what some of the other television channels offer.

Often diarists would talk about watching a program because there was a lot of discussion about it both in the media and in their communities. Furthermore, they watched programs because they were recommended by others. Such recommendations might come from friends, family, or from editorials and television critics. Recommendations are especially influential for films and one-off television programs, where there is little prior knowledge of the program.

Planning is Often Short Term

Predominately, people used the guides to map out the coming evening’s viewing or while sitting in front of the TV to consider what was available. Longer-term planning was focused on programs users particularly wanted to watch or record (such as a film or documentary).

The Design Concept

We developed a low-fidelity mobile prototype to support viewing planning in three ways:

By providing a recommendation facility where one user can send suggestions to another’s mobile, and where “recommender services” (such as broadcasters) can also send alerts. Suggestions are sent in a text message mode that alerts the recipient as soon as one is sent.
By focusing the mobile guide on personal highlights for the user by default listing only the upcoming day’s highlights. Users can create lists of their favorite programs and program types.
By reminding them of a program they indicated they wanted to watch or record.

PDA with a tv schedule on it — Figure 3. The paper-and-PDA mock-up used in evaluation.

Evaluation

We developed and facilitated a proof-of-concept study with twelve users (five female, seven male, aged 18-35). Each participant worked with the prototype, which consisted of mock-up dialogue screens attached to a hand-held computer manipulated by the investigator in a naïve Wizard-of-Oz form. That is, the prototype had no built-in interactivity, but was animated by the investigator choosing what to display next. We asked users to work through a series of tasks that demonstrated the extended program guide facilities. At the end of their session, we asked them to rate the features and provide additional feedback during an interview.

We found that the participants saw the personalized TV highlights as the most useful feature. The reminder function was second in popularity. In terms of the recommendation features, users expressed a strong preference for family/friend recommendations over third-party ones. Even so, many participants reported that they would prefer to phone or text someone, rather than explicitly recommend a television program. There were also concerns about being alerted each time a new recommendation arrives. Participants saw a less intrusive system, allowing recommendations to be dealt with when it is convenient to do so, not necessarily when they arrive, as preferable. That is, a system much more like email than texting. Television programs that viewers are currently watching provide the only instance where recommendation alerts are desirable. However, most participants still reported that they would rather call because there is no guarantee that their friend or family member is watching television at that moment.

photo of a woman holding a PDA — Figure 4. A participant using the prototype during an evaluation.

Conclusions

We have presented some initial arguments and evidence that suggest conventional electronic program guides need augmenting. Program guides do not support planning away from the television set, and do not support the engaged, social style of interaction required in some TV planning scenarios. Migrating some of these activities to a handheld device appears to be a promising approach that merits further investigation.

While the social element of the system—the personal recommendations—was viewed favorably, our study suggests that there is much further work needed to understand how to integrate existing collaborative viewing practices (such as word-of-mouth promotion) within emerging iTV technologies.

Engaging with Mental Health: Opportunity for Collaboration

June 15, 2011 No Comments

With several notable exceptions, much of the early research on the use of technology was justified on the basis of increased access, such as electronic contact for a natural extension of face-to-face dialogue, and the computerization of self-help materials. Increased engagement and improvements in the effectiveness of treatment have received less attention. Collaboration between human-computer interaction (HCI) and mental health professionals can help to maximize the effectiveness of new technologies. While mental health professionals have the necessary domain expertise, HCI professionals are experienced in design methods and are likely to have a broader knowledge of the potential uses of new technologies. For example, the experience of HCI researchers is important given the high cost of systems failures in sensitive interventions. Other ongoing areas of HCI research, such as designing for personal reflection and behavior change, can play a valuable role in future research on mental health technologies.

Examples of Exploratory Systems

In our own research we have primarily focused on the design of interactive systems to support talk-based, or psychological, approaches to mental health treatment. Over the past five years we have developed several exploratory systems:

Personal Investigator

Personal Investigator (PI) (see Figure 1) is a 3D computer game designed to support adolescent mental health interventions. It incorporates Solution Focused Therapy, a goal-oriented, strengths-based intervention approach. PI is used in clinical sessions involving one therapist and one adolescent. It aims to ease the difficulties many adolescents experience in engaging with face-to-face treatments. Two clinical evaluations of PI have been conducted, the first with four adolescents, the second with twenty-two. Results indicate that PI can have a beneficial impact on interventions, supporting improved client-therapist relationships and improved client engagement.

screencap from computer game — Figure 1. The 3D computer game “Personal Investigator” is designed to help adolescents in face-to-face treatments.

PlayWrite

PlayWrite extended the ideas developed with PI. Rather than providing a fixed game, it provides a flexible game template and enables end users—mental healthcare (MHC) professionals—to create and adapt the content delivered through this template. Using PlayWrite, MHC professionals have created ten games, implementing different theoretical approaches to MHC and targeting a range of difficulties including depression, anxiety, and anger management. For example, gNat’s Island supports a six-week cognitive behavioral therapy (CBT) intervention for depression.

Mobile Mood Diary

Therapists, particularly those practicing CBT, often ask clients to complete paper-based mood charts, but adherence can be low and they can provide unreliable information. Mobile Mood Diary, a mobile phone- and computer-based mood chart system (see Figure 2), was developed to make recording moods more convenient and reliable. A controlled school study was run studying compliance variation between a paper-based chart and the Mobile Mood Diary. An initial clinical evaluation with ten adolescent clients has also been completed. Results showed a high level of adherence amongst participating adolescents over a sustained period.

Design Challenges and Future Directions

A more detailed description of previous work in mental health technologies is available in Computers in Talk-Based Mental Health Interventions by Coyle et al, and Hands-on Help: Computer-aided Psychotherapy from I. M. Marks. We conclude this article by considering several significant challenges for future work in this area.

Theories for Design and Evaluation

Mental health technologies remain a relatively unexplored design space. Alongside exploring the potential of a wider range of technologies, there is a need for more detailed investigation of theories, from both the mental health and HCI domains, which can help to generate ideas and support us in reasoning about designs. For example, mental health frameworks such as the Skilled Helper Model provide a structured overview of a helping intervention. Behavior change models such as the Trans-Theoretical Model may also prove useful, as could activity theory, which has previously been applied to a range of healthcare settings. Applying models for complex medical evaluations may also prove beneficial.

Strategies for Collaborative Design

While collaboration with domain experts and end users is important in many design areas, it has a particular significance in mental health areas. The ethical requirements and stigma associated with mental health means that designers often face severe restrictions on access to people experiencing difficulties and to situations in which interactive systems will be used. Faced with such constraints, HCI practitioners may have to rely on the expertise and insights of mental health professionals. However, research in related healthcare areas has shown that the success of interdisciplinary teams is not something that can be taken for granted. We have discussed initial strategies for collaborative design decision making in our paper “Clinical Evaluations and Collaborative Design: Developing New Technologies for Mental Healthcare Interventions.” We need to build a deeper understanding of collaborative design strategies, and develop techniques which support effective interdisciplinary teams.

Adaptable Systems

Therapists often work with clients from different socio-cultural backgrounds, experiencing a range of disorders of varying severities. Furthermore, teams of therapists often have different theoretical backgrounds and adopt a variety of eclectic approaches to working with clients. In such situations, fixed systems offer limited benefits. Systems where the core capabilities are reusable across a range of situations, and which can be adapted to the needs of end users, would be more useful. PlayWrite represents an example of such a system. In this case, mental health professionals play an active role in adapting the system. This approach can also help to support collaboration.

Support for Resource Allocation Strategies

Better targeting of resources and needs-based service provisions have become important goals in mental health service delivery. Models such as Stepped Care encapsulate this notion, with distinct steps—intensities of treatment—ranging from inpatient treatment, through primary care teams, dealing with issues such as mild depression to general practitioners supporting initial contact and diagnosis. Designing interactive systems to support these different levels of care, and transitions between them, represents a major challenge.

The Truth is Out There: Using Mobile Technology for Experience Sampling

September 15, 2008 No Comments

As practitioners, we spend much effort designing and testing products and services within the confines of our offices and labs when we know that a rich user experience lies outside. We need more research “in the wild,” where people use the very interfaces we take so much time to design, test, iterate, and develop.

Now, what if we could conduct studies “in the wild” using a robust platform that collects data from the users, their actions and environment, and even handles compensation, recording, and event and task scheduling? And what if this tool was not an additional research device that must be carried around, but a device that users have with them wherever they go— their own mobile phone?

User Experience is More Than “Walk-Up-And-Use”

Traditional usability testing involves task-based research in the lab where designs can be tested, iterated, and validated. This methodology is ideally suited to assess usability in a highly tactical manner where specific design questions can be answered, and is critical to the user-centered design process as it ensures that the core features are usable. However, the specificity of usability testing is also a limitation because it mostly focuses on the “walk-up-and-use” user experience. Exposure to the product is limited, and participants generally don’t interact with products they actually own. While this method is very useful for design iteration, it lacks external validity and does not lead to an understanding of how the product is used in the real world. Ultimately, true product usability, usage, and usefulness should be determined in a natural environment over time.

“Remember When You…” Methods Don’t Quite Capture It Either

The data collected to describe the real-world user experience are often obtained through surveys or focus groups. While these methods are beneficial in the early-stage assessment of feature importance, pricing, or intent to purchase, results often tend to be high-level and less helpful when specific design decisions need to be made. Even when these methods are focused on design questions, the collected data are largely retrospective. Asking users to reflect on a task completed in the past is not as effective as asking the same questions during or immediately following the completion of the task. So, we need research methods that can tap experiences as they happen.

Getting Out in the Field Can Be a Challenge

Ethnography, or observing behaviors in a natural environment, avoids retrospection and potential confabulation associated with “remember when you…,” leading to higher validity of the collected data. However, the study logistics can be complex, and fieldwork and data analysis time-consuming.

Longitudinal Research is a Major Endeavor

Traditionally, longitudinal research involves data collection over time. Instead of tapping the user experience just once, this method allows for repeated testing on the same set of users over multiple sessions. It can assess learning based on how the user adapts to and uses the product during critical periods of the product’s lifecycle.

Longitudinal research is also compelling because it provides information on how the user experience can change over time. Users can learn how to use the product more efficiently, can lose interest in the features that they initially found appealing, or find another use for the product. These findings give us insight into product usage and applicability to user needs. The methodology can also be influential to design as it fills in the “post-walk-up-and-use” gap that is left open following typical usability testing.

The longitudinal approach, however, is used infrequently, despite the advantages it offers. At CHI 2007 (ACM’s conference on Computer-Human Interaction), a new special interest group on longitudinal research was formed. Out of the fifty-plus participants, less than a quarter had conducted a longitudinal study in the past. At CHI 2008, the group discussed possible reasons for the dearth of this type of research. The main reasons revolved around complex logistics, time-consuming data analysis, and the perception that a longitudinal study is a major endeavor.

Experience Sampling Can Tap Motivation

Traditional usability testing captures the “walk-up-and-use” aspect of the user experience, ethnography helps us understand the user experience in the real world, and longitudinal research provides insight into learning and usage over time. However, we still lack information on the drivers for usage. Influencers that impact usage of a product include motivation, fun, and emotion.

This is where longitudinal research using experience sampling methods can help. Experience sampling refers to in situ research where a phenomenon is examined in the place and time it occurs. A common example is a “pager study” where the researcher “pages” participants at different times and prompts them to provide information via a diary at that very moment. These data, while time-consuming to extract and analyze, provide deep insight into user activity, motivation, and other cognitive and social dimensions. They can identify patterns of use, context that drives usage, and motivators.

Leveraging Advances in Mobile Technology

We need to be able to capture experiences in the places and at the times they occur, but in a more efficient manner, while retaining the tactical and rigorous elements of in-lab research. Mobile technology has advanced to a point where researchers can do more than simply page users and ask them to make a diary entry. We can now expand upon longitudinal and experience sampling techniques to better solicit, monitor, and receive data on user interactions at given points in time. And the mobile device itself can be much more than a conduit between the user and researcher.

Setting Triggers to Capture Specific Experiences

An interesting aspect of this technology is that the user’s own actions can trigger data collection. User phone activity is automatically and remotely monitored. When an event of interest occurs (for example, the user accessed a particular place on the phone or physically
entered a building), a prompt requesting data on the experience is sent to the user. In an attempt to avoid interruptions, the system tries to predict when the experience is finished or when the user is “free.” Thus, the research program actively solicits feedback on the experience at a time when the user is more likely to be available.

In the example shown in Figure 1, a person uses a navigation application on a phone to find directions. If the application is of interest to the research, the user’s actions are recorded and sent to the research lab. When the application is closed and the phone has been idle for a period of time, the user is asked questions regarding his/her experience.

illustrations of senarios that qualify as Event-Triggered Testing — Figure 1.

Improving Task-Based User Testing

Using this new technology, we can conduct large sample usability studies of phone functions and features (such as adding a new contact to the Address Book) with the user’s own device and in his/her natural environment. Mobile service providers have data on how often and when users make calls or send text messages, but their vision does not extend much further. They do not know if the user called John by using speed dial, by selecting his number from the Address Book, or by entering the number. They do not know if the user even entered John into his or her contacts.

In the example shown in Figure 2, a researcher sends a prompt to the user to complete a specific task on their music phone (for example, create a playlist). The user completes the task and then answers questions about the experience (ease of use, frequency of use, preferences, satisfaction, etc.). All data are sent back to the researcher, including data from the device itself to assess if the task was successfully completed and how it was completed (for example, screens and clicks).

Assuming that novices had difficulty with this task during in-lab testing, we could retest it on owners of the phone after a few months of ownership. This would help us determine whether or not they were able to learn the functionality over time.

illustrations of scenarios that qualify as Task-Based Testing — Figure 2.

User Feedback — More Than Just Text

Free-form user feedback is now a reality with QWERTY keyboards on mobile phones. Anyone who observes today’s youth texting incredibly quickly can envision the tremendous data collection benefit over handwritten diary input that requires transcription and review. The mobile device can be leveraged to open up novel forms of data collection never before possible without specialized equipment. For example, the phone can display questions and solicit feedback in the form of text, single or multiple selection, or slider manipulation. It could present images for the user to provide context for questions. Imagine the depth of experience that could be captured if the user could respond by speaking their responses, taking a picture, or recording a video of what they are doing. The capabilities of a mobile device as a research tool create a wealth of opportunities.

Capturing Non-Phone Experiences

We could also use the phone as a data collection method to capture non-phone experiences. Consider a pharmaceutical scenario to study drug treatment adherence and patient experience when using medication. The phone can prompt the user to take their medication and provide feedback on their current symptoms. In the case of a medical device such as a lancet or injector, the user could provide feedback on the interaction with the device and the level of pain they experienced when using it.

Another example could be the setup of a home theater product. It is not practical to test in the lab the myriad of home theater configurations that exist in homes today. Practically, lab testing can only test a few common configurations, but we know that many problems can occur when the user attempts to install a new entertainment component into the nest of wires and components that make up his or her home entertainment system. Using mobile technology, hundreds of home environments can be tested to assess setup difficulties. Figure 3 depicts a user installing a new HD receiver. The user can use his or her mobile device to report on the usefulness and usability of the instruction manual. The user can be prompted to use the camera to take pictures of their home system configuration as well as document any problems encountered during setup. The ability to collect data on hundreds of users using this technique is compelling.

Illustrations that qualify as Capturing Non-Phone Experiences — Figure 3.

Next Steps

As practitioners, we should strive to constantly improve our user experience research techniques with the goal to uncover how users really use products. Several methods using mobile technology have been used in the past. These include solutions using SMS messaging to solicit and collect data or even questionnaires on old Palm Pilots. However, these methods involve manual work by the researcher (such as sending SMS prompts). The need for a more robust system where research parameters can be managed automatically is clear. The technology described in this article can make research easier through device- and server-powered research solutions that can be configured to reside on today’s mobile phones.

We have developed a new mobile technology platform called LEOtrace® Mobile that would make all this possible. It is currently available on devices running Windows Mobile 6 with a release schedule that includes Symbian, Linux/Java, and other operating systems in late 2008 and 2009. The platform is in beta testing with users now and it is sufficiently advanced to automatically trigger prompts to the user based on events such as the start and close of applications, when applications move to the foreground or background, when a call is initiated, received, and ended, when a text message or email is received and sent, or even when a specific website is visited or when the user walks into a particular GPS location (for example, a shopping mall).

The potential of remotely capturing user experiences as they occur is a more ecologically valid approach than many of the traditional techniques we have been using. The method easily scales to “large N” studies and makes research more efficient. When coupled with objective user behavior data collected from the device itself, the user’s subjective feedback can be analyzed to more effectively improve product design and must affect a positive change in the user experience.

Understanding User Adaptation to New Technology in the Manufacturing Industry

March 31, 2019 No Comments

The manufacturing industry has existed for more than a century and has time-tested processes. Today, this industry is officially in the age of Internet of Things (IoT) adoption. As discussed in a white paper by Schneider Electric, this change should be seen as an evolution, not a revolution. Companies have invested hundreds of millions in industrial automation and control systems and are often unwilling to invest hundreds of millions more to replace those systems with new technologies.

A photograph of an industrial plant. — Figure 1. For manufacturing to adopt Internet of Things technologies, they have to evolve their existing processes.

There is a need to leverage modern technologies to accommodate changing conditions. Overall, the industry is understanding this need in order to keep up with changing times, to optimize their processes, to increase operational efficiency, and to deliver good experiences to customers.

Some companies are motivated to invest in IoT technology to comply with government regulations, such as monitoring pollution and providing live data. Any failure to meet these regulations can be lethal, as it can lead to the company’s license being revoked. Considering these circumstances, Internet of Things has emerged as a new technology that helps improve processes by providing automation, control, and efficiency, thereby saving time and money.

However, adopting IoT in this industry has many challenges, especially in developing countries. These challenges range from the employees and their awareness of technological products to the manufacturing environment of existing industry practices. Manufacturing plants are often located in rural or semi-urban regions away from cities. And most of their non-management employees, such as operators and floor managers, live in close-by villages and towns. The Internet has touched their lives; however, the use of the Internet is still limited to communication and entertainment.

Using the Internet to integrate IoT-enabled devices for purposes that have higher stakes—such as in an industrial plant—is still a novel practice for these companies and sometimes involves applying traditional methods for the practice to gain acceptance. As technology adoption in manufacturing increases, the growing skills gap is also becoming increasingly apparent.

Challenges in this Space

A photograph of a machine operator working while sparks fly. — Figure 2. Scene of a workshop.

When companies began making IoT based software products for the manufacturing industry, they started facing many challenges—for example, lower user adoption and users abandoning the product after the purchase. This created a desire to understand the reasons for these behaviors and to find solutions to fix them.

This industry also has multinational corporations that are working toward cost savings through automation. These corporations also have big budgets for training and experimentation. In the same market, there are owner-based companies or small and medium enterprises (SMEs) with smaller budgets that do not wish to invest in intensive digitalization for their manual practices, and subsequently do not train their employees in the adoption of the IoT. However, these companies may be forced to take this route due to government regulations.

In the paper “Cultural Barriers in the Adoption of Emerging Technologies,” the authors assert that SMEs that are likely to adopt new IT usually have a CEO who has a positive attitude toward innovative technology and is knowledgeable about IT. However, even this group struggles with technologically unskilled employees. Thus, while designing an enterprise product for a desktop or mobile app, it is important to discuss the challenges in adopting IoT technologies for this user group.

Product-Based Challenges

The challenge begins right after the purchase of the product: from installation of the product, getting started, day-to-day usage, and calling customer care and resolving issues to periodically upgrading the software. The following are the top challenges in adopting technology for this user group:

Environment: Some small-scale manufacturing companies do not have or have limited access to the Internet in their plants. This makes it difficult to download large files or to engage in live chat sessions with customer support. After installation, poor Internet connectivity may make the system unstable, requiring constant human intervention.
Installation: Software installations are complex, as they require defining many parameters unique to the plant and government regulations for that particular industry. In addition, there is often no specific role of an IT admin in some manufacturing plants. Many times the individual responsible for installing and using the software may perform many roles related to plant management. This person often comes from a manufacturing background and is not trained in the complexities of IT or the installation of plant software. Therefore, close assistance or help for installation of the software is required.
Getting started: Enterprise software often lacks a good onboarding experience to help users learn about the product. The importance of using a digital product versus previous approaches may not be communicated by management. What little learning material is available is often in text-heavy documentation that is hidden under layers of a UI. This causes user frustration and lowers motivation to invest time to learn about and use the product.
Day-to-day usage: Because companies often use complex interfaces and terminology that is for power users such as IT admins, when users start to use the product, they are often unable to discover features that are hidden in complex navigation, and are burdened with unfamiliar concepts such as reporting dashboards or configuring settings.
Upgrading software: Software builds are constantly evolving. New versions need to be downloaded, and systems need to be upgraded. Going online, downloading, and upgrading the working system can create a fear in users that a change may break down the mission-critical software.
Trust: For ages, companies have been using manual systems; trusting that a machine can give reliable information requires a significant mindset shift. At the same time, these systems are evolving, so they are subject to errors and system failures. This adds to the existing distrust of a system.

Language-Based Challenges

Language also plays a vital role in learning and adopting these systems. Documentation and support are often provided in English only; we need to remember that in several countries English is used for only official work. English-educated users within the same organization can use it effortlessly, but there is another group of workers in the field who have difficulty comprehending it. Most on-the-ground workers have a limited understanding of English and can read only regional languages, but much of the documentation and software produced by industries are typically in English. This makes it harder for users to read documentation, follow instructions, and to seek help. This leads to many problems, including higher support calls that require customer support representatives who speak the local language to help resolve issues.

Importance of UX Research in the Manufacturing Space

In order to create effective software for this population, we need to create products based on an understanding of the users’ needs and constraints. Therefore, user research is critical.

There are two important factors to this issue. First, user research helps identify all kinds of users, both primary and secondary. Second, users play a very important role in performing tasks, as well as planning time and resource management. After identifying user types, a strategic research approach needs to be designed to gain user insights. User profiles assigned by product managers often focus on primary users only, as they focus on product usage without considering alternative manual approaches. Thus, educating internal teams becomes a part of the research strategy to help them take a more holistic view. Transparency and trust are critical to get information from the teams.

Conducting the Research

When conducting research for an IoT-based software product that is used by industrial plants, it is very important to understand all types of users, their roles, responsibilities, and how they are associated with the product.

Data Collection

User groups can vary from power users to users who might be only using a basic Android phone for limited operations. In a blog post by Dr. Rishi Bhatnagar, chairperson of the Institution of Engineering and Technology—IoT Panel for India, he noted that “IoT requires multidisciplinary skillsets and a combination of domain knowledge, which is scarce in current workforce.” Therefore, it becomes critical to identify and recruit the right participants.

Furthermore, within the company it is important to study the employees in customer-facing roles and to gain an understanding about their challenges and their perspectives on the challenges for their users. Overall participants should be both internal and external:

Internal refers to customer-facing employees, like product managers, sales teams, and customer support.
External refers to all kinds of stakeholders, from end users (who use and maintain IoT enabled devices) to the purchasing manager, the IT admin, and so on.

Process

The study should start with gathering information to understand internal stakeholders’ goals and perceptions. The next phase should be about gaining insights about the end users to help form the research strategy. These insights will help determine the best method to use, what usability metrics should be used, and so on.

Phase 1: Research with internal stakeholders.

Before meeting with end users, it is important to understand the needs of the internal team. This involves identifying all the internal stakeholders and studying the hurdles they face with their customers. The research method can use a mixed approach by using in-person interviews, listening to customer support center calls, and attending a few sales sessions with the sales team.

Phase 2: Meet with users.

There are two groups of users: One is management and the other is the workers. The first group is from the top management that might be focused on getting high-level information to see how the plant is doing and how efficient workers are. The information gathered during this phase will help these top managers optimize the company to produce a better product.

For the second group of users, the plant workers, getting insights from this group needs to be carefully planned, using designed questionnaires and scripts. It is very important not to rely on self-reported user feedback but to validate as much as possible by visiting physical worksites and observing how people work. The following are a few methods that work particularly well with these types of users:

Interviews: Speak with management users; they are the people who have defined measurable goals for the organization and therefore drive product adoption. Beginning the study by getting insights from this group is a good starting point. The information gathered during these interviews will highlight issues such as the company’s perception about users’ challenges and will also help to inform on-field strategies.
Contextual inquiries: This later stage involves visiting users in their environment, such as visiting factories and studying workers and management while they are using the product. During these sessions, users are asked questions and then observed while they work in their own environment.
Diary study: While studying employees once at work in the plant can provide information concerning usability challenges, this one-time observation will not expose their day-to-day usage, how they learn to use their tools, and the challenges they face producing a product. To get more insight on these aspects, a couple of weeks of study can help provide this necessary information. One important tool to record this insight is the use of a diary. During the contextual inquiry sessions, you may consider asking employees to submit a diary for the study. The employees could submit entries daily using a program like WhatsApp. Entries could include pictures, videos, or audio recordings because simply typing their submissions may be considered tiresome, discouraging this user group from submitting entries.

A photograph of two women sorting sticky notes on a wall. — Figure 3. Researchers analyzing in the lab.

Findings

The findings report should include a variety of insights, including feedback from top management, limitations of customer support and sales, design/usability issues, and other insights. Therefore, it is important to include both a high-level summary for everyone and department-specific detailed reports that can be presented to each stakeholder group. There should be a way to track the changes and the criticality of the issues in each relevant department. It helps for teams to understand what is lacking in which department and who is working toward fixing it.

Summary

The benefits of IoT adoption for the manufacturing industry are significant. However, the rate of growth is still low, and only 30% of manufacturers are investing in digital transformation, while the rest are held back by outdated business models and technology.

Therefore, designers have an opportunity to create solutions that can address these challenges. User research can be the first step to enable teams to get insights to change the product strategy and improve the user experience with a product. For workers, implementing a new technology creates significant challenges as they adapt to a new way of working. As researchers and designers, we can help create products that facilitate better working conditions and provide more accessible technology for a less privileged section of the world.

Search Results for: diary studies

Finding the Right Tool

Translated Transcript from Figure 6

Advantages of Diary Methods

Studying Temporal Dynamics

Reporting Events and Experiences in Context

Determining the Antecedents, Correlates, and Consequences of Daily Experiences

Limitations of Diary Studies

Designing a Diary Study

Eliciting Diary Entries

Involving Users

Pen and Paper vs. Electronic Diaries

Diary Structure

Collecting and Analyzing Data

Example of a Before-Usage Diary Study

Quantifying UX Over Time

The Snippet Technique

What Is a Diary?

A Word about Nomenclature

Why Use Diaries?

Which medium Is best?

Paper

Digital

Diary Exercises to Try

What about platform?

Exercises

Timeline

Photo Album

Moments

Easy & Hard

“Love is…”

Deprivation Day

Final Words of Wisdom

Recruiting Older Participants

Establish relationships with directors of senior housing facilities

Reach out to those who care for older adults

Use your social and professional network

Interacting with Older Adults

Support the effects of aging in written communication

Support the effects of aging in verbal communication

Gain their trust

Be on time

Be patient

Be prepared to probe

Older Participants’ Involvement in Research

Older participants were highly committed to the research

Participation benefited the older participants beyond compensation

Compensation was still important

Final Thoughts

What is a Probe?

Why would you use Probes?

Challenges using Probes

Examining and Using Probe Data

Probes for Industry

Background

User Research

Pain Diary

Research Methods

Findings

Naming and Branding

sentoms Icons

sentoms CP

Overview

Design Process and Final Prototypes

Future of the Project and Broader Applications

Electronic Program Guides

Multiple Device User Interfaces

Paces of Interaction

Television Planning Diary Study

Planning Can be a Social Activity

Viewers Look for More of What they Like

Planning is Often Short Term

The Design Concept

Evaluation

Conclusions

Examples of Exploratory Systems

Personal Investigator

PlayWrite

Mobile Mood Diary

Design Challenges and Future Directions

Design Challenges and Future Directions