Skip to content Skip to footer

Search Results for: diary studies

Little Red Sprite Learns Finance: An Exploration of Mobile Diary Studies Using Avatars and Chat Apps

In this article, we share considerations and lessons learned from using WeChat as a mobile diary that can be applied to future studies using chat apps.

A mobile screen with the login for WeChat
Figure1. WeChat as a mobile diary.

The mobile diary is a great remote research tool to help uncover the daily lives of consumers and gather field-based consumer insights, especially as naturalistic data can be difficult to obtain through focus groups, interviews, or observations. This is particularly true when presented with geographical and temporal constraints. In a typical mobile diary study, participants record their reactions and responses to the subject matter (for example, brand communication, shopping activity) while on-the-go, on their own mobile devices, the very moment they experience them.

Our Singapore-based research team used an existing chat app, WeChat, as a diary tool during a series of mobile diary studies that focused on understanding how Chinese consumers go about their banking and financial activities in China. Use of an avatar rather than conventional human facilitation allowed us to engage more authentically with consumers and gather richer insights.

Finding the Right Tool

If we disregard China as the context (think firewall, app usage restrictions, and language barriers), there are a plethora of mobile diary tools available, ranging from paid to free, from custom apps to good old email. Custom mobile diary apps like dscout, ethos, and indeemo can be useful in research as they not only help collect information but also allow basic coding/tagging or grouping, enabling researchers to see data clearly.

Collaborative tools like Evernote and Google Docs also enable researchers to receive real-time updates from participants as they record their activity in text, photos, and/or audio.

However, for our study we wanted to leverage existing behaviors of how people communicated with loved ones, akin to documenting their lives as they chatted with their friends and families. This led us to considering chat apps as a tool. We wanted to avoid having participants purchase or experience potential difficulties using a new mobile app, especially since our participants lived within a different culture and used a different language than us.

Reducing the time needed for participants to grasp the technicalities of how to use a mobile diary study tool allowed us to devote more time to explain the value of the research and encourage their buy-in. Based on our previous experience, participants were willing to be more open, and, in turn, provide more valuable insights when they realized their recorded financial experiences could help banks provide them better services.

WeChat was selected for its ubiquity in China, with a reported 750-plus million users monthly, allowing us to benefit from participants’ familiarity. Other popular chat apps in Asia like Google Talk, Facebook Messenger, WhatsApp, Korea’s KakaoTalk, and Japan’s Line Messenger were disqualified due to China’s firewall and data restrictions.

In addition to chatting about their financial behavior using text, participants were encouraged to use pictures and voice messages. Some preferred recording audio while on the go, especially some older participants who seemed to be more comfortable with voice messages. Voice messages proved to be useful in that we could hear more contextual data, too. For example, we could hear the change in tone of voice if the participant was in a hurry or a hushed voice if speaking at home with a baby present.

Although the name WeChat suggests a chat focus, it also has a social media feature, allowing users to share posts with selected friends on their individual walls. Gaining access to users’ social media walls after participants added our WeChat account as a “friend” provided researchers a sense of overall lifestyle, allowing for cross-referencing wall postings with data collected from their diaries. Similarly, participants could refer to the objective of the study and download related content from the researcher’s wall.

Calendar even screen (in Chinese)
Figure 2. Gaining access to users’ WeChat walls provided researchers a deep view of the lifestyle and behavior of participants, allowing for cross-referencing with data collected from their diaries.
Calendar event screen (in Chinese)
Figure 3. Participants found information on the objective of the study and downloaded related content from the researchers’ WeChat wall.

Lastly, though not an initial consideration in selecting WeChat as the medium, the payment features on WeChat largely aided participants in understanding the nature of our study. Following the success of WeChat Red Packets (微信红包), which mimics the Chinese tradition of gifting money in physical red envelopes called hongbao (红包), Tencent has put considerable focus into WeChat Pay (微信支付), which can be used to transfer money between WeChat users (peer-to-peer) and make payments online and with participating offline retailers.

Leveraging familiarity (participants were all WeChat users), we referenced WeChat Pay features as examples of financial activities to record and discuss, on top of other common examples like visiting a bank or paying for groceries.

Screens in Chinese as described in the caption
Figure 4. Screenshots from participants using payment features on WeChat. From left to right, WeChat Pay for taxi services, WeChat Pay for merchant purchases, a WeChat Red Packet message, and a WeChat Red Packet containing digital money as a gift.

Someone to Talk To: Avatar as Researcher

Our initial challenge with the mobile diary was that participants assumed the diary facilitator understood the nuances underlying the recorded activity and bypassed cultural explanations. For example, a participant posted, “I spent 47yuan on taxi today,” but stopped short of explaining how exactly the payment was made. It was only via post-study interviews that we found taxis are commonly hired via mobile platforms such as DiDi in China, instead of flagging one off the street. Another challenge was that many were protective about personal financial habits or sharing such information with a stranger-researcher from a foreign land.

Given the social and conversational nature of WeChat, we turned to designing an agent to act as our face—our representative and voice. For our purposes, such a character needed to be friendly, approachable, culturally appropriate, and act like a curious outsider.

Two cartoon-y icons
Figure 5. Alibaba’s Taobao (online shopping site) Mascot 淘公仔 Tao Doll (left) and Tencent’s QQ (mobile messaging platform) Mascot QQ企鵝 QQ Penguin (right).

At the beginning of each study we introduced a curious cartoonish alien we named Little Red Sprite (小红精灵) wanting to understand ‘Earthlings’ to encourage participants to document in more detail, explaining their activities and all relevant Chinese cultural aspects–just as they would do for an interested non-local. Along with designing the agent to look endearing, we evaluated the mascots of China’s Internet giants and made Little Red Sprite’s conversation with participants more colloquial and playful.

A screen with the mascot and a message in Chinese
Figure 6. Our mascot, Little Red Sprite, saying hello and explaining the mobile diary study.

Chatting Is What a Chat App Is About

Each study was 30 days long; with time participants’ posts started to be simpler, less frequent, and questions like, “Am I doing this right?” began appearing. We learned how crucial it was to close the loop while verifying our interpretations. To elicit a higher quality and quantity of posts, we leveraged WeChat’s sticker culture to engage in dialogue and reply to posts with customized stickers to appear more personable. The role of Little Red Sprite was critical here to prevent participants from being intimidated or overwhelmed by the requirements of the research. We believed an advantage of this type of avatar was in its social capability, essential to the continued dialogue. Care was taken to have Little Red Sprite display the emotions of gratitude and curiosity, as opposed to a tone of disappointment when posts were not made.

Also, participants were interviewed directly on WeChat posts, such as, “You mentioned heading to the supermarket with your credit card. Were there specific benefits of using a credit card?” Or, “I noticed your posts hardly mention shopping online. Do you usually shop online?” Dialogue not only demonstrated the kind of reciprocity that was inherently motivating, but also helped in drawing a detailed picture of the activity.

The relationships built up over the course of the diary study were also essential to the success of follow-up interviews conducted. The trust and familiarity gained by participants enabled intimate sharing as opposed to a rigid Q&A. Dialogue with participants on WeChat helped to scope questions for the follow-up interview phase, prompting in-depth questions like, “Could you explain why you have different credit cards and bank accounts for shopping online and shopping offline?”

Little Red Sprite, the agent, seemed to resonate well with participants and their responses were open and friendly, especially during the follow-up interviews. Participants addressed the researchers as “Little Red Sprite” and apologized through WeChat for not spending money or not posting. These were responses we had never received from previous (non-agent) studies! Their posts also had more explanations and included participants’ comments like, “Oh yes, you might not know, but in China giving your partner access to your bank account is a sign of trust. It’s common for girls to handle her boyfriend’s finances.”

Conclusion

Financial studies are often tricky because they involve sensitive data. Not everyone is open to sharing how much they spend, what they spend on, or how much they earn.

While our study was designed for the Chinese market and used a chat app locals were familiar with, the same principles can be applied to other markets and cultures where there is a high penetration of social chatting apps.

Participants’ understanding of how their responses were of value encouraged them to contribute more expressively and regularly. Enticing participants to continuously post about their day required vigilance and creativity in both feedback and encouragement.

The overall success of the technique was measured by the higher quality data (the amount of detailed and personal information) provided by participants, and by engagement (how frequently the participants posted). For some, posting even continued after receiving notice that the study had ended. The relationships developed (with Little Red Sprite) led to the success of the interviews, as trust and familiarity enabled more open, rich, and detailed conversations.

The study was made possible by NCR and we’d like to thank all of our research participants.

[greybox]

Translated Transcript from Figure 6

Hi, I’m Little Red Sprite from Planet Ingot and I currently work at the customer experience department. I discovered a book about Earth in the library the other day and saw that there was a country called China that’s filled with lots and lots of people. I was so excited and I’m sure there’s a lot to learn about how people on Earth experience banking. Unfortunately, I couldn’t find much information on my own, could you help me understand how you bank? I would like to learn and introduce better ways of banking back to Planet Ingot. Thanks for helping!

[/greybox]

 

Dear Diary: Using Diaries to Study User Experience

Keeping a diary is not just a hobby undertaken by teenage girls trying to make sense of life and love. Diary methods have been widely used in medical research where patients are asked to keep a diary during a clinical trial or disease treatment. In Human-Computer Interaction, a diary study is a qualitative technique for collecting data on what users have done or experienced. Much like a travel journal contains descriptions of the traveler’s experiences, a UX diary contains descriptions of the user’s experiences with a system.

What kind of UX research questions could diaries help answer? Depending on the design of your diary study, you may get information on the overall impression of a specific device, usage of features, technological acceptance, emotions associated with task performance, or learnability of an application. Of course you could also get this information with a survey, an interview, or a user test. But only diary methods will give you access to temporal and longitudinal information gathered in a natural context of the interaction.

Advantages of Diary Methods

Studying Temporal Dynamics

As stated in “The User Experience White Paper,” there are several time spans of user experience depending on the moment of usage (see Figure 1). Typical methods used to study UX shed light onto only one kind or moment of the UX. Surveys are often related to Episodic UX: users are asked to assess the interaction after usage. User testing helps collect data on Momentary UX. You may also have an idea of Anticipated UX by conducting focus groups.

Figure 1. Time spans of user experience. (Source: “The User Experience White Paper,” Dagstuhl Seminar on Demarcating User Experience, 2010)

The main advantage of a diary study is that it allows collecting longitudinal information. Let’s imagine you would like to assess a new mobile application. The first part of the diary study may be administered prior to usage by asking users how they imagine the application, what they would expect from it, and if they have any prior experience with similar applications. Then, you might ask them to report their everyday impressions and feelings during usage. Finally, during a debriefing interview or a post-study questionnaire, you may ask the users to reflect on their experience after usage.

Reporting Events and Experiences in Context 

Another issue we should be always mindful of is the impact of context on the results. For example, when conducting a user test or an interview, the presence of the researcher may bias the results, leading to a positive evaluation of a system despite poor user performance. This is called a social desirability bias. When we know we are being watched, we will tend to behave in a way we believe is socially acceptable or desirable.

As researchers, we should want to capture life as it is and not how it looks in a controlled setting. The emergence of mobile technology especially highlights this need to study use “in the field” (see Figure 2). That’s when the diary methods come in handy, as they focus on reporting events and experiences in their natural context.

Two photographs of people using a mobile phone: in an office and on a construction site.
Figure 2. On the left: A participant interacting with a mobile application in a usability lab. On the right: A participant using the same application, but in a natural environment. Do you really think that the user experience is the same in both situations?

My company developed a collaborative environment to address the needs of architects and engineers in the construction sector. The mobile application was tested in a usability lab, and flaws were identified and fixed. But once used on a construction site, there were more problems than expected. Effects of dust, noise, brightness, and handling were underestimated during both the design and evaluation steps. Moreover, collaborative issues in this real on-the-go setting were quite different than those simulated. The subsequent use of a diary study helped improve the system by revealing real user issues and needs in the context of their workplace.

Determining the Antecedents, Correlates, and Consequences of Daily Experiences

As mentioned before, UX is rooted in different time spans, each one being influenced by another. Momentary UX is colored by Anticipated UX in the way that user’s expectations have to be met or exceeded to lead to a positive experience. Then, Momentary UX will be distorted by the filter of cognitive processes when changing into Episodic UX and Cumulative UX. Expectations, mindsets, moods, and social or physical contexts all color user experiences. A diary study captures these influences, shedding light onto how the UX in each time span has formed.

For example, findings from diary studies can reveal that it was the bad mood of the user or brand criticism heard on TV that impacted the user’s assessment of the interface. You may also discover that the game you are evaluating provides enjoyment only when the user interacts with close friends or family, even if the functionality supporting this was neglected by designers. Finally, you may notice that a single flaw you considered as minor discourages the user and leads to system abandonment.

Limitations of Diary Studies

Nothing is perfect! As with most methods, you need to look at the pros and cons of designing a diary study. The main disadvantages of the method are the cost and time associated with:

  • Participant recruitment. The quality of the results depends on the participants since the diary is bound to the expressive ability of the writer. This is especially true for a study with a lot of open-ended questions. You also have to achieve a high level of participant commitment to obtain sufficient and reliable diary entries.
  • Training or briefing sessions. A diary study often requires detailed training sessions to ensure that participants fully understand what needs to be reported, how, and when.
  • Data analysis. Analyzing diary entries is time consuming. This activity takes even more time in the case of a pen-and-paper diary.

Designing a Diary Study

Eliciting Diary Entries

There are three categories of diary protocols based on the way in which entries are elicited:

  • Interval-contingent protocol, in which participants have to report their experience at regular predetermined intervals (for example, every two hours or every day).
  • Signal-contingent protocol, which uses a signaling device to prompt the participants to make an entry.
  • Event-contingent protocol, which requires participants to report each time a specific event occurs.

Protocol selection depends on the purpose of the study and activities under investigation. The interval-contingent option, although widely used, does not help participants remember to make diary entries, which may lead to missing data. Signal-contingent protocols overcome this drawback but are intrusive. An event-contingent protocol might be used especially when you are interested in studying the experience created by a specific event, such as a system error or notification. Rate and timing of self-report have to be set up according to research needs. Do not be too demanding or your diary will become a burden to your participants; a maximum of two to three entries per day should be enough.

Involving Users

One of the major challenges in diary studies is maintaining participant commitment at a high level to ensure diary completion. You have to make sure that participants understand the scope and descriptive depth needed for the diary entries. During the study, you can remind participants of the importance of diary completion. A cash or gift incentive, or the possibility of keeping the assessed device, can be very helpful. However, be careful not to base the incentives on data quantity or you may get tangled up in irrelevant information.

Pen and Paper vs. Electronic Diaries

Because UX studies are often related to the use of technological devices, the choice of an electronic report is often judicious. It simplifies data analysis and gives the opportunity to combine diary entries and log files (for example, date, time, and task performed). E-diaries also provide participants with innovative forms of reporting. Imagine a diary study through voicemail entries, text messages, video, or pictures!

However, you should always try to adjust the method to the target users. If you are conducting a study with elderly people, a pen and paper diary will probably be more appropriate. Your participants should feel comfortable with the process of keeping a diary.

Diary Structure

Diaries may be open format (users record activities and events in their own words) or highly structured (where closed-ended questions are pre-categorized). Do you wish to encourage general reflection or need precise information? A mixed approach is often adopted to collect both qualitative and quantitative data. Do not forget to add a clear set of instructions on how to complete the diary, and stress the importance of recording events as soon as possible after they occur. You may also use non-verbal tools to facilitate completion (see Figure 4), especially for specific target users such as children or people with a disability.

A series of 5 cartoon faces indicating a range of emotions through the mouth (turned down to turned up) and other features.
Figure 4. The Self-Assessment Manikin is a non-verbal scale to measure emotions.

Collecting and Analyzing Data

After designing your diary, do not forget to pilot-test it before the study launches. Participants should keep the diary for a maximum of two weeks, as people’s commitment quickly decreases over time. If the diary is electronic, you should monitor its completion for each participant. Do not hesitate to contact participants during the study to ensure that everything is going well.
At the end of the study, conduct follow-up interviews with your participants. That’s your opportunity to have them explain some of their entries. A good understanding of participants’ qualitative answers is required to correctly interpret the data.
The analysis of diary entries depends on how structured the diary is. Quantitative data may be analyzed using any statistical analysis software. Dealing with qualitative data is more challenging but certainly worth the effort because these will provide you with valuable information on subjective experience.

Example of a Before-Usage Diary Study

A diary study of rendezvousing (described in the Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work) examined how people meet up, with the goal of understanding how technology could support them.
Martin Colbert used a diary method because the events of interest were “too rare, private, and geographically dispersed for direct observation, and users too easily forget important details of their behavior to report them accurately long after the event.” Participants had to report their rendezvousing behavior for fourteen days. One diary entry was made for each rendezvous event. Each entry was composed of an open-ended, narrative description in the participant’s own words of what happened and why, and the participant’s responses to a rendezvous questionnaire, which asked for specific details of each event (see Figure 3).

View of the form
Figure 3. A diary entry participants were supposed to complete following each rendezvous.

The researchers used the results to extract the overall structure of rendezvousing and to build typical rendezvousing scenarios. Eight possible causes of problematic rendezvous were identified, including transportation problems, over-running of previous activities, poor planning, lack of travel information, lack of geographical information, lack of information about other rendezvousers, spontaneous additional tasks, and failure to value success.

The study helped identify what kind of position-aware service was likely to be most useful and how it could be designed to shape rendezvous experiences.

Quantifying UX Over Time

I have used the abridged AttrakDiff tool (www.attrakdiff.de) to evaluate the hedonic and pragmatic quality of a mobile application (see Figure 5). By asking participants to report their feelings and impressions using a short survey, I was able to draw a UX chart representing the evolution of UX over time (see Figure 6).

Mobile screen with survey
Figure 5. AttrakDiff survey on a mobile phone.
Graph showing change in scores over time.
Figure 6. AttrakDiff scores illustrating how the UX of a mobile application changed over a period of ten days.

The Snippet Technique

The snippet technique developed by the Stanford HCI Group aims at overcoming the limitations of diary studies performed under mobile or active conditions, where there is no time to make thorough entries. Participants capture text, MMS, or voice snippets when on the go and later annotate the snippets on the web (see Figure 7).

Figure 7. Snippets captured during the day serve as memory aides for full diary entries in the evening.[
即使对于有经验的从业人员,研究和评估用户体验也是一项有挑战性的工作。我们不是超级英雄,也没有水晶球,怎样才能找到一种方法,通过考虑所有与用户体验相关的因素,来获得站在用户角度的体验感受?

用日志方法来研究用户体验具有很多优势。本文介绍为什么要使用日志以及如何设计一个日志研究。作者对最适合采用这些方法的研究问题、可以根据日志研究的发现做出的设计决策、进行研究所需的工具,以及分析日志报告的方法提供了实用指导。

The full article is available only in English.UX를 연구하고 평가하는 것은 경험 있는 전문가를 위해서도 어려운 과제입니다. 슈퍼히어로가 되지 않거나 수정 구슬 없이 UX와 관련된 모든 요소를 염두에 두고 사용자의 관점으로부터 경험을 얻어낼 수 있는 방법을 어떻게 찾아낼 수 있을까요?

UX 연구를 위한 일기 형식 방법의 장점은 상당히 많으며 본 논문은 왜 당신이 연구를 위해 일기를 이용해야 하며 어떻게 일기 형식 연구를 설계하는지 설명합니다. 저자는 이 방법에 가장 잘 맞는 연구 문제, 일기 형식 연구의 연구 결과를 바탕으로 할 수 있는 디자인 결정, 연구 수행에 필요한 도구 및 일기 형식 리포트 분석 방법 등에 관한 실용적인 지침을 제공합니다.

The full article is available only in English.O estudo e a avaliação da experiência do usuário é uma tarefa desafiadora, mesmo para profissionais experientes. Sem ser um super herói ou ter uma bola de cristal, como podemos encontrar uma metodologia capaz de capturar a experiência do ponto de vista do usuário, levando em conta todos os fatores relacionados à experiência do usuário?

As vantagens dos métodos que utilizam diários para o estudo da experiência do usuário são numerosas e este artigo descreve por que você deve utilizar diários em sua pesquisa e como projetar um estudo utilizando diário. O autor fornece uma diretriz prática com relação a questões de pesquisa mais adequadas para esses métodos, decisões de projeto que podem ser tomadas com base nos resultados de um estudo utilizando diário, ferramentas necessárias para conduzir a pesquisa e formas de analisar os relatórios do diário.

O artigo completo está disponível somente em inglês.UXの調査および評価は経験豊富なプロでも容易なことではない。スーパーヒーローであったり、魔法の水晶玉を持たない限り、どうすれば、UXに関連したすべての要因を考慮しつつ、ユーザ視点からの体験を把握できる調査方法を見つけ出せるだろうか。

UXの調査における各種ダイアリー法のメリットはいくつも挙げられるが、この記事ではなぜ研究においてダイアリーを使うべきなのか、またダイアリー調査をどのようにデザインすべきなのかについて説明する。筆者は、これらの調査方法に最適なリサーチクエスチョン、ダイアリー調査での結果をもとに行えるデザイン決定、調査を実施するにあたって必要なツール、ダイアリーのレポートの分析方法について、実用的なガイダンスを提供している。

The full article is available only in English.Las ventajas de los métodos que utilizan diarios para el estudio de UX son numerosas y en este artículo se describe por qué debería utilizar diarios en sus investigaciones y cómo diseñar un estudio que incluya diarios. La autora ofrece consejos prácticos respecto de las preguntas más adecuadas en estos métodos de investigación, las decisiones de diseño que se pueden tomar en base a los resultados de un estudio con diarios, las herramientas necesarias para realizar la investigación y las formas de analizar los informes que incluyen diarios.

La versión completa de este artículo está sólo disponible en inglés.

Dear Diary: Getting the Most from Respondent Journals and Diaries

During a recent study, I asked respondents about their snacking: when and why. You would have thought that these respondents were among the healthiest on the planet. Why, the apples they consumed could have denuded an entire orchard! Luckily, though, I had asked the respondents to keep a mobile diary of their snacking behavior for the week prior to the focus group. How quickly these respondents had forgotten the chips, Twinkies and, yes, even the Philly cheesesteaks that comprised their snacking behavior over the past week. The client received rich insights into aspiration vs. reality, and I had a lot of fun in the process.

Diary research can lead to deeper insights from your respondents. But creating a great diary assignment is more than just passing out blank paper and having respondents note every time they make a cup of coffee or use their online banking app.

What Is a Diary?

Diaries have three basic elements that, taken together, separate them from all other qualitative data:

  • They are respondent generated.
  • They capture a specific behavior, activity, experience or feeling.
  • They happen over time.

Diaries can be given as homework (see Figure 1), but they do not have to be assigned prior to other research. They can work great as a standalone exercise or when assigned after an initial phase of research.

Figure 1. Diaries vs. Homework
Figure 1. Diaries vs. Homework

A Word about Nomenclature

In my work, I sometimes use the words “diary” and “journal” interchangeably. In his book Let’s Explore Diabetes with Owls, writer David Sedaris — himself a prolific diarist — noted, “Often, the terms are used interchangeably, though I’ve never understood why. Both have the word ‘day’ at their root but a journal, in my opinion, is a repository of ideas, your brain on the page. A diary, by contrast, is your heart.”

Indeed, there is something very personal conveyed by the term “diary,” which may be exactly what your study needs. If, however, you are asking school-age boys to keep some kind of diary, consider calling it a “journal” instead. I found from personal experience in my own house that “diaries” tend to elicit eye rolls in this demographic.

Why Use Diaries?

Diaries allow us to extend the amount of time that we spend with respondents. Furthermore, diaries allow us access to respondents at times when we might not want to be with them or when the researcher’s presence might compromise the respondent’s ability to be candid.

Diaries help researchers explore three broad types of question:

  • Usage: Often, a marketer wants to know how the company’s product or service is actually being utilized in the real world, to identify problems or opportunities. Diaries kept over time allow respondents to capture their actual experiences with the client’s product in detail.
  • Process: Diaries are great at reflecting consumers as they go through a particular experience, and they often provide deep insights about the purchase process. Consider asking your respondents to keep a diary when they buy a car, apply for college or go through chemotherapy and to note the expected and unexpected steps they take along the way.
  • Experience: Sometimes, marketers have broader questions that they need to answer to help drive innovation or marketing messages. Experiential diaries help answer the question, ‘What is it like to…?”

Ultimately, diary studies are powerful for clients because they tend to lead to previously undiscovered consumer needs, which then point to opportunities for clients to innovate (See Figure 2).

Figure 2. Diaries lead to opportunities
Figure 2. Diaries lead to opportunities

Which medium Is best?

Believe it or not, paper can be a fabulous medium for the right type of diary study. Digital, of course, is rapidly expanding our ability to be with consumers 24/7, so it is important to think about what best fits your study.

Paper

Paper diaries provide a wealth of opportunity to ask respondents to think and work creatively. For example, a paper diary can allow respondents to write, draw and illustrate in ways they would struggle to do with a digital medium. A major drawback of paper diaries, however, is that you do not get to see what a respondent is working on in real time. If there is a problem with your instructions or if you wanted diary data to help drive discussion-guide development, you are out of luck.

Digital

More and more, the smartphone is becoming the digital-diary medium of choice; however, respondents can also participate in digital diaries using their PC or tablet. The pervasiveness of digital tools makes them particularly attractive for diary exercises: Since most of your respondents are probably carrying their mobile phones with them at all times, it makes good sense to ask them to use those devices to complete a diary exercise. That said, mobile technology tends to favor shorter, less introspective responses than one might get with, say, paper or a diary that uses a PC-based platform where respondents can type at length.

Table 1. Pros and Cons of Paper and Digital Diaries

Type Best for Drawbacks
Paper Diaries Smaller budgets

Creative assignments

Low-tech respondents

Easy access to diaries during follow-up research

No research access during data collection

Hassle factor – printing, shipping, collecting, scanning, etc.

Digital Diaries Capturing data at the time it is happening

Engaging clients

Learning about respondents before they get in the room

Pre-screening respondents for follow-up research

Can feel sterile – need to work to create a warm environment

May not be a fit for all respondent types

Cost not usually free

Diary Exercises to Try

Ready to take your diary exercises to the next level? Below are a series of six activities to try that go beyond asking respondents to jot down every time they do X or purchase Y. All of the activities provide a basic structure, but they still leave room for you to customize and improve them to meet the needs of your next study.

What about platform?

A number of these exercises are best executed on a digital platform. The platform providers change all the time; for an up-to-the-minute guide, I would recommend checking out the New Qualitative Guide published by the Qualitative Research Consultants Association (QRCA) or, better yet, talking to QRCA colleagues who work with digital platforms regularly.

Although many of these exercises can be accomplished using free platforms, tools developed specifically for the marketing research industry make it really easy for moderator, client and respondent to participate in ways that are seamless and appropriate. Sometimes spending a few extra bucks on a paid platform is worth the investment in terms of your time and sanity.

Exercises

Timeline

Often, a diary seeks to capture a particular behavior over a time period that may vary by respondent. For example, a consumer in the market for a new car may research, shop and test drive for weeks before making a choice, or he may walk onto the lot and make a purchase rather quickly. The Timeline exercise is simple. Give each respondent a large piece of paper with an arrow drawn across it to represent time. Each time (s)he does something related to the activity of interest, (s)he notes it on the timeline. In this way, behaviors are captured as they happen but don’t need to follow a specified format.

Photo Album

We know consumers are already taking pictures of every aspect of their daily lives. Why not harness that for marketing re- search purposes? A photo-album diary exercise goes beyond having respondents take photos by asking them to organize those photos into a meaningful story with rich captions. And when the study is over, you can cherry-pick the best albums and use one of the many online tools to actually create hard-copy albums that the client team can use and refer to for months and years to come.

Moments

When we want to know about the emo- tions surrounding a particular behavior, the “moments” diary exercise is ideal. Consider the difference between “Take a photo of your morning coffee” and “Keep a journal of your coffee moments.” Ask respondents to tell you as much as they can about the moment — what was going on, how they felt and what made it a moment.

Easy & Hard

Often, clients want to capture product usage so that they can figure out what is working well and what is not. In these cases, consider structuring your diary exercise to capture what is easy and what is hard about using a particular product or service. In fact, this exercise works well with any polarity: How about yum/yuck, best/worst or fun/boring?

“Love is…”

Sometimes, it is important to understand how consumers think about a big attrib- ute such as love, clean or healthy. But getting them to articulate what these things mean can be tough. (How do you define clean?) A diary that asks respondents to capture photos that encapsulate a particular attribute can quickly get to higher-order benefits and provide agency or creative teams with a ton of fodder for creative development.

Deprivation Day

Deprivation day is not an exercise unto itself, but rather something to add on to a longer diary assignment. The way it works is this: If you are having respond- ents log some activity or product usage for a defined time period, ask them to spend a day going without that activity or product. For example, if respond- ents spend a week capturing “coffee moments,” ask them to go without coffee for a day and journal what hap- pens. What did they eat or drink or do instead? How did it feel? (You might call these “Non-coffee moments.”) A word of warning: Tell respondents dur- ing the recruit that you are going to ask them to do this; some might not be able to go without.

Final Words of Wisdom

Whether you are doing your 1st diary study or your 100th, you are never alone in the process. The technology providers all have relevant guides and case studies to spark your imagination, and I have found that many of them are willing to give you a free trial to get you started learning their platforms. After all, your success is their success! So, why not add a diary activity to your next project, and see what you can learn?

Conducting Field Studies with Older Adults Lessons for Recruiting and Testing Older Users

Last winter, we ran a usability study in our lab with adults aged 75 and above. This age group is often classified as the “older-old” population, as opposed to the “younger-old” population of 60-75. We had done considerable research with this older-old population before, and we knew to expect higher-than-average cancellation rates because of participants’ health and transportation issues. But when a light snow wreaked havoc with our testing schedule, we decided to take the research to our participants. In doing so, we discovered that field testing and field research work especially well with older adults.

We’ve conducting over 100 interviews and field tests with adults 75 and older in the past two years, mainly in the area of consumer products and home healthcare devices.

Recruiting Older Participants

One of our first challenges, as in any study, was finding the right participants. For us, this meant people aged 75 and older with a variety of health-related issues who would allow us to conduct interviews in their homes. In recruiting this population, the following strategies worked well for us:

Establish relationships with directors of senior housing facilities

We found that senior housing facilities and assisted living facilities gave us great access to potential participants. Focusing on these facilities also cut down on the travel time between interviews. When we explained our study to the directors at the centers, they often became great advocates for our research. They circulated information for us and, in some cases, even arranged for us to speak to their residents to explain our studies. This allowed us to clarify the purpose of the research and reassure residents that we were not selling anything. Having someone they trusted vouch for us helped our participants feel more comfortable with our studies.

Reach out to those who care for older adults

Paper flyers and internet postings that targeted caregivers, as opposed to older adults themselves, proved very effective. Again, this provided older adults with the reassurance that someone they trust-often a son or daughter-had learned about the study and found it legitimate. When working through caregivers, however, it is important to ensure that the actual participants themselves have agreed to participate and are not being pressured by their caregivers.

Use your social and professional network

Networking was an effective way to locate older adult participants. For example, one of us had a late cancellation for a study that involved an international client who was leaving the next day. At the last minute, she was able to replace the participant with the mother of the contractor who was working on her house!

Our early round of interviews also provided excellent networking opportunities. Our first participants told their friends about the study, who then wanted to participate in later rounds of testing. However, this method has potential pitfalls. For example, we encountered a group in one senior housing center who took down posted flyers because they wanted to reserve participation for their circle of friends.

Interacting with Older Adults

Maintaining clear and open communication while we interacted with older-old participants contributed to the success of our studies. Being trustworthy, patient, and punctual also enhanced our interactions during the sessions.

Support the effects of aging in written communication

To support the visual decrements of aging, we increased the font size in our flyers and used bold text. We also needed to support the effects of aging on cognitive functions such as memory. We made flyers as straightforward as possible, with simple sentence structure and no unnecessary details. In a diary study, we provided a list of clearly stated reminders, labeled, “Important things to remember,” for participants to reference on their own. Consistency also proved to be important. In an early study, our informed consent form had the line for the signature first and the line for the printed name second, while our cash receipt form had the order reversed. When we presented the second form, some of the oldest participants signed the Name line and then were embarrassed when they read “Signature” on the next line. In subsequent studies, we fixed the forms to present the name and signature in a consistent order.

two different forms
Figure 1. Inconsistency between signature lines on forms can cause problems.

Support the effects of aging in verbal communication

We were able to support hearing impairments by speaking loudly and clearly, and in some cases, participants asked us to sit on the side nearest their “good ear.”

As with written communication, we strove to be clear and concise and to avoid long, detailed instructions. Asking older adults to retain many details at once taxes their working memory, which becomes especially vulnerable with age. We chunked the information, stated each point clearly, and left time for questions between topics.

Gain their trust

Older adults are often targets for scams, so it was important for us to be open about our research and to answer all questions, even if we had to repeat ourselves. During a recruiting presentation at one assisted living facility, we were careful to state that we were not selling anything and that we would pay honoraria for participation. Still, in the end, some residents asked how much it would cost for them to participate in our study.

We limited our team to one or two interviewers, so that we would not overwhelm participants, and made sure to tell them ahead of time how many of us to expect.

We also invited caregivers to be present for interviews, both for their peace of mind and for participants’ reassurance. One caveat, however, is that sometimes caregivers, who feel very connected to their loved ones and want to help, may answer for the participant. In these cases, we gently redirected the question, saying, for example, “Now, Mary, what do you think?” We were careful to ask for participants’ permission before recording or taking photographs. We asked them during the screening call and again at the beginning of the interview. We used portable video cameras to record usability tests. For field interviews, we took digital photographs and handwritten notes.

Be on time

Arriving on time is very important in establishing participants’ trust. Our participants were often anxious for us to arrive at their homes. To ensure on time arrival, we allowed extra travel time to account for unforeseen delays, such as traffic. Participants often expressed appreciation for our punctuality and found this gesture considerate and courteous. It set a positive tone for our sessions.

interviewing with video camera
Figure 2. Equipment for recording usability tests in the participants home.

Be patient

As busy researchers, we were aware of all that we wanted to cover in our interviews. But our participants treated us like company in their homes and, as guests, it was important to allow time for them to tell us about their families, and not to jump right into the interview questions.

In answering questions, participants sometimes needed to give us more detail than we really needed. For example, we might ask how long they lived in their current residence, and they would answer with a story about when and how their spouse died, how their children worried about them remaining in their family home and encouraged them to move into the senior housing facility where they currently lived. Paying attention to these stories not only showed respect for our hosts, it gave us valuable insight into our participants’ lives, including their use of our clients’ products.

To account for this deep level of communication, we allowed about 50 percent more time for these interviews than we would with a younger population.

Be prepared to probe

Probing is a standard interviewing technique, but with older adults, we sometimes found it necessary to probe around even simple statements. For example when we asked, “How is your general health?”the majority said, “Good.” However, further probing indicated some had had cancer, heart surgery, respiratory disease, or other ailments. For these participants, ìgoodî seemed to mean “Good for someone my age,” or “Good, considering all I’ve been through.”

In probing, we were careful not to be intrusive, which can be a delicate balance. While we wanted to know of any health concerns that affected our research, we did not want to elicit medical information that was personal and/or confidential. When one participant offered to show us all his medications, we politely declined.

schedules on a refrigerator
Figure 3. One participant’s schedule of activities on the refrigerator.

Older Participants’ Involvement in Research

As we interviewed older participants, we learned more about how they related to the research in general and how they understood their participation in our studies.

Older participants were highly committed to the research

Our participants took their involvement in our studies very seriously. Only one of sixty participants in our diary studies dropped out-and she did so because she was hospitalized. Participants were also highly motivated to complete their diaries accurately. To help aid their memory, some carried note pads and recorded activities as they occurred so that they would not forget. We have conducted similar diary studies with younger participants and did not find nearly this level of commitment.

Participation benefited the older participants beyond compensation

A number of participants told us that being involved in the research was a positive experience for them because it made them feel needed and valued. Several said they were pleased to be able to do something that would help others in the future. The research also appeared to stimulate some older participants because it encouraged active engagement in a task. One of the facility directors reported that she felt that our diary study actually “exercised” the memories of residents at her facility.

Compensation was still important

As with other studies, we paid our older participants at the end of each interview. While a number of participants told us they would have been happy to participate “for free,” they clearly appreciated the honoraria we gave them. Compensation not only supplemented their income, but it helped express the value we placed on their time.

Older participants are each unique and vary in many different ways There is huge variation among persons of any age in personality, as well as cognitive and physical functionality. We saw participants of the same age who could hardly walk and others who exercised for an hour or two per day. While some participants had little activity beyond watching television, we took pictures of one 93-year-old’s calendar and refrigerator door, which indicated a life full of activities and responsibilities. We met an 88-year-old who still tutors teens in a city school.

We were amazed at how frank and open our participants were in our interviews. They sometimes told us things they wouldn’t tell their children–that they took naps in the middle of the day, for example. One woman told us that she was looking forward to going to her next home. We asked where she was moving, and she responded “Heaven…I hope.”

Final Thoughts

Our fieldwork with older-old adults opened our eyes to their needs and their strengths. By being trustworthy and adapting our communication to meet their needs, we were able to recruit participants and conduct successful field studies.

As with any study, we became sensitive to the issues facing our target population. With our older participants, we found ourselves looking ahead at our own future and wondering how we will fare. Thankfully, our work in this area has the potential to positively influence the lives and health of current and future senior citizens.

Positioning Field Studies for Company and Customers

Getting Started Within Your Company

The mantra for usability practitioners is “know your users”. Similarly, when proposing field study research, it is important to understand who you are “selling” field study research to—who are the people holding the key to unlocking the resources, schedules, and budget required for the research activity to occur. Understanding this audience allows you to create persuasive arguments to gain their support and keep them interested and engaged in the discussion. It also increases the likelihood of obtaining resources and financial backing for your project.

The task of gaining support for field study research may be formidable depending on the scope of your project. Large-scale field study initiatives may require support from multiple levels in the organization (for example, directives from upper management executives, support from middle management, and cooperation from individual employees,) and from numerous divisions within the company.

When proposing field study research to executive management, first identify their priorities and discuss the benefits within the context of these priorities. For instance, if executive priorities are centered on customer satisfaction and its effect on the bottom line—sales and revenue—then position field studies as a mechanism to build and foster the customer relationship. If the priority for upper management is operational effectiveness, discuss how field study data can streamline product development by reducing ambiguity, providing clear and concise requirements early, and eliminating churn and indecision during development. Additionally, remember that upper management has limited time and a large organization to manage. Keep the conversation at a high level, but be prepared with details if asked. Try to leave the executives with the answers to what you want, the cost (in time, resources, money), and the return on investment for the company. Chances are they won’t give you an immediate answer to your request, but if you leave them with a clear sense of your proposal, they are likely to respond favorably.

When proposing field studies with product managers and marketing managers, think about their primary responsibilities and concerns; they need to specify and prioritize product requirements to meet the customer and market needs. Be sensitive rather than antagonistic about how they determine requirements. Offer field studies as a complementary method that primarily yields user requirements regarding interaction designs and usability criteria rather than business and functional requirements. For example, if the requirement is to have email functionality, the field study research would provide insight on users’ current experiences and expectations for email, which would help specify users’ functional and interaction requirements. Field study findings provide necessary information that will later facilitate discussions and mitigate conflicts during development and construction of the product. The data also helps with prioritizing requirements and specifying designs to ensure a highly usable, quality product.

Engineering teams have the burden of building the perfect product. They want to ensure the product they commit to building is truly meeting customers’ needs. Include engineering in field visits; it provides engineers with a rare opportunity to directly observe users and see how their products or similar products are being used and implemented. It offers a glimpse into whether the product is being used as intended, how it has been extended and improved, and any shortcomings. This insight is invaluable and a good reality check for engineers; the value in field studies is well-defined product requirements and use cases, which leads to less churn on product specifications and user interface designs. Engineering is always faced with schedule pressures, thus is sensitive to factors that may extend timelines and supportive of measures to minimize schedule disruptions. Field studies should be positioned as requiring time early in the schedule, but providing a depth of data that reduces time spent revising requirements and design specifications during the construct phase. Additionally, if scope changes are necessary as the project progresses, field study data helps inform decisions about the trade-offs and consequences to the user.

Getting to the Customers

Now that you have gathered support from inside your company, it’s time to identify and recruit customers to visit and observe. With B2C (business-to-consumer) products, it is a matter of determining the objectives and contacting consumers directly to arrange for visits. With B2B (business-to-business) products, identifying and recruiting customers may be more involved; you may need to convince partner consultants, third party vendors, and customers before gaining access to end users to observe.

As with all research activities, involving the appropriate participants is crucial. Participants need to address your research goals to ensure their input is valued. You do not want anyone to question the findings because the proper user group was not studied. Depending on the nature of product release, there are different user types you want to observe. For example, if the product is entering into a new industry, then the users would be non-customers. If the product is being enhanced and sales revenue is expected from current customers upgrading to the new version, then you want to target those customers. You will need to work with different people within your company to access the various user types, and you will need to develop a value proposition for each to gain access to their customers.

The sales team builds and enhances customer relationships for future sales deals, especially with high revenue generating customers, or those with potential for an upgrade or cross-sell. Field studies should be positioned to the sales teams as a tactic for fostering customer loyalty and a demonstration of individualized customer care. Provide sales with examples of positive perceptions customers will have following the field studies.

If the primary focus for the next product release is refining the existing product and addressing known issues, tap into the customers through customer service and technical support. They are well aware of the product shortcomings and have direct access to customers experiencing the most difficulties. Pitch field studies to customer service and technical support organizations as a mechanism to improve product quality and increase customer satisfaction.

Customer Value

Regardless of whom you work with to determine customers for site visits, it is important to create a value proposition for the customer. Place yourself in the position of the customer and think about the questions customers would ask: Why would the field study be important to me? How is it a valuable use of my time and my employees’ time? What are the benefits to me? Access to customers’ employees is not easy as there may be perceptions that you would be intruding on their employees’ privacy and time. Customers may fear the field study would distract employees or reduce productivity. Be sure you understand the customers’ perspectives and come up with compelling benefits before contacting the customer. For example, participating in a field study allows the customer to voice their requirements early on and is an opportunity for the customer to make their employees feel empowered because they can be involved in the definition and design of the products they use. Again, it is important to develop messages that align with the customers’ core values and alleviate their concerns. Be upfront and clear about expectations, time commitments and duration of the activities, and determining deliverables the customers may expect in return for their time and participation in the field study.

Analyzing and Communicating Findings

The ultimate goal of research is to make a difference to the product design. The best way to assure findings are applied is to give people data they’re interested in. Do not overwhelm them with the minutiae. Analyses and reports need to be tailored for the various recipients. Avoid the academic style of traditional reports, and make the report memorable and easy to comprehend. Focus on getting a few key points across to the appropriate recipients. Present the findings using formats appropriate for each group. Use marketing terms, (“Top Ten Findings”, “Myths and Realities Exposed”) to make your points memorable.

Observational data may be analyzed to provide a list of improvements they can make to the current implementation of the product. This provides some short term, easy fixes for immediate improvements. Additionally, customers may be interested in seeing a summary of how the latest version has resolved some of their issues, and which issues the company should consider when upgrading to the next release.

For the user experience team, the data are analyzed to produce the usual deliverables, such as user profiles, task analysis, wants and needs analysis, and user interface design requirements.

For the sales team, the results may lead to targeted selling. It’s possible to provide the sales team with specific selling points to encourage customers to upgrade (a summary of customer’s current issues may help them sell an improved version) or provide insight on customer needs and where an additional product may be offered to fill that need.

For the cross-functional product team, which includes—but is not limited to—product managers, marketing managers, the engineering team, quality team and documentation, field studies provide real customer usage scenarios for prioritizing requirements and determining solutions. A detailed context of how users interact with the product provides a vivid view of the users that is indispensable during product design and development. It provides data that may be referred to throughout the product development cycle to make informed decisions and to understand the consequences to the end user when making trade-offs.

Conclusion

Field studies provide benefits to many groups—users, managers, sales, marketing, and support staff. By understanding and communicating these benefits, you can maximize your ability to undertake such studies and the effectiveness of the deliverables they produce.

Cultural Probes: Understanding Users in Context

One of the latest techniques in the Human Computer Interaction (HCI) community is a data collection mechanism called cultural probes. These were introduced in 1999 by William Gaver, an academic based at the London Royal College of Art, in the publication Interactions.

Since then, cultural probes have been discussed, developed, and used extensively in academic HCI research. Although industry practitioners are starting to adopt cultural probes, there are still many questions around the usefulness and practical application of this technique. This is not a simple question and, as Gaver says, “The use of probes is geared toward design generation, not problem solution.”

What is a Probe?

So what exactly is a cultural probe and how would you explain it to your skeptical boss in the elevator? Gave describes cultural probes as “designed to prompt and elicit information from people about their lives and local culture.” A simple explanation is that it is a diary maintained by a user over several days or weeks. Typically, the diary includes photographs taken by the user, pictures cut from magazines, voice recordings, and written notes. The technique is also called “diary study,” “media diary,” and “photo collage.”

To create their diaries, users are given a “pack.” The contents depend upon the objectives of the data collection, the characteristics of the users, and the imagination of the designer, but common items are scrapbooks, pens, cameras, and diaries.

Some of the more innovative and interesting items that have been used in packs are catch phrases printed on sticky labels and simple digital memo-takers that are repackaged as dream recorders. These let users record ideas or potential requirements as they occur to them, such as when awakening from a vivid dream.

Users are briefed on how to use the packs and what information to record. They are asked to create the diary independently, in their own environments, with regular entries over an allocated time. The designer may contact or visit the users occasionally to monitor progress, answer questions, and keep the users focused.

photograph of notebooks with pictures and post-it notes attached
A typical cultural probe

Why would you use Probes?

All data collection techniques have advantages and disadvantages. To understand why probes might be useful, we need to consider the reliability of data constraints on time and money and the need to collect data for non-work or non-office activities—for example, the use of mobile devices for social, leisure, or on-the-road usage.

Reliability of data is a fundamental challenge for all designers. Every data collection technique risks unreliable data. For example, focus groups may drift so far from reality that the descriptions of actions bear little relation to actual behavior. In laboratory usability tests, the lack of real context may distort behavior. In contextual enquiry, it may simply be impossible to control observational conditions due to the noise of other activities unrelated to the project at hand.

Cultural probes are an additional tool with a new set of strengths and weaknesses. For example, the reliability of data may be enhanced by users being experts in their own lives. Cultural probes provide the opportunity for users to document their lives in their own context, with minimal interference from the designer. Probes also provide the opportunity to triangulate against another set of independent data (such as focus group, usability tests, or survey findings) and thus improve reliability.

Time and money constraints may be circumvented. Commercial organizations often expect HCI practitioners to somehow have an inbuilt and expert understanding of the target audience, and they may make little allowance for the time-consuming and expensive process of collecting and analyzing data to gain a significant understanding of the audience. Because cultural probes place the onus of the actual data collection on the user, practitioners can focus their effort on the analysis of data.

Access to interaction and behavior for non-work activities, or work performed outside the traditional workplace, is a new challenge for commercial HCI. The traditional work environment tends to be well-defined and accessible. Work within this environment is generally routine and structured. People are often able to describe their work, working environment, and patterns.

However, the contexts of both non-work and leisure time activities are much harder to define, and any data collection technique requiring the designer to be physically located with a user may disrupt or destroy the quality of the activity. Furthermore, the lack of routine and structure in these activities may greatly expand, or make uncertain, the amount of data collection required. This is one of the most significant advantages of cultural probes. Data can be collected easily despite the unstructured, non-routine, and easily disturbed context of the activity. Users frequently become very engaged in the studies, and often comment that keeping diaries and scrapbooks is fun.

Challenges using Probes

As with all data collection techniques, probes have challenges. The most significant of these is ensuring that users understand and maintain focus throughout the data collection process. This is not trivial, as probes are very much a hands-off data collection technique; the user is neither observed nor directly guided during the day-to-day creation of data.

On the other hand, the designer must be careful not to be too restrictive or prescriptive and thereby lose ideas, inspiration for design, and unexpected information about user behavior and interactions. The risk is that a lack of focus may lead to very noisy data that is difficult to analyze and does not provide useful input to a specific design. Probe studies should include an initial briefing session and at least one interview visit or telephone call to the user during the early part of the diary-keeping process. This is extremely valuable as many probe users need to be focused, re-focused, or simply taught how to use the pack materials.

Despite the reduction of time and effort required from researchers (since users collect their own data), probe studies consume elapsed time, which may be in short supply in tight project schedules. Therefore, probe studies should be run as early as possible in projects.

Researchers must also take care to gather an appropriate amount of data. The number of users, the length of the data collection period, the probe pack materials, and the requested frequency of diary entry must all be taken into account in the design of cultural probes. For studies in which I have been involved, about twenty to thirty minutes per day was asked of users. This may vary, with some days where less or no data is entered into the diary, and other days where more data is entered.

The length of the data collection period is also important. For example, one study involved seven weeks of data collection but five weeks would have been enough. Some of the users lost interest, finding that some of the data contained nothing new and there was simply too much data. At the other end of the scale, in an industry study for a specific product, the challenge was determining the minimum amount of time to make the study worthwhile. We tried seven days and found that to be just enough. This was mainly due to the simple and easily definable area of focus for the study.

Another influence on the quality of data is the level of interest and participation of the users, as they are not only the source, but also the transcribers of the data. Users may volunteer, ideally with the interest and personal motivation to support diary filling over an extended period. Alternatively, if you need to closely match a target audience persona, or have practical concerns such as time constraints, you may need to recruit and pay users. This is usually done through agencies. In this case, you must take care to maintain the users’ interest, and remuneration must be carefully considered.

Examining and Using Probe Data

Probe data is not straightforward to use. In the first place, probes can generate tremendous amounts of data, requiring significant time even for a first reading. The more complex question, however, concerns how to examine and analyze the data. Academic researchers have spent considerable effort trying to understand how to interpret the data from probes. Gaver comments in one study that, “Our probe results are impossible to analyze or even interpret clearly because they reflect too many layers of influence and constraint.” However, in another part of the same study he comments that, “Sometimes the trajectory from probes to designs is relatively straightforward, and design ideas can clearly be traced back to probe returns.”

There is currently no one standard method for using the data from probe studies to influence design. The most straightforward method is for the HCI practitioner to brainstorm design ideas. The data can provide a rich insight into the lives of users.

Probe data also generally provides a good understanding of a real-world domain. The data enriches the ideas of the designer or researcher and provides a supplementary influence rather than a direct prescription for design solutions. In particular, probe study data, together with interviews and debriefing, provides an excellent source of data from which personas can be constructed.

In one project, probe data was used to create scenarios which, in turn, were used as the basis for participatory design sessions. The data informed and enriched the scenarios, providing excellent context and motivations. The participatory design personnel included researchers involved in the study, HCI experts, and the probe users. It was found that the probe users, while having an intimate knowledge of their own data, did not have the HCI skills for designing, and that the best design concepts came from HCI experts who were familiar with both the probe data and the context of the project as a whole.

Probes for Industry

Cultural probes are a new technique worthy of attention from commercial practitioners. They are particularly useful for novel non-work design situations where user behavior is relatively unknown or difficult to access. Probe data can be very noisy, however, and the data at times may be incomplete, unclear, and biased. It is important to set expectations that the data will enrich the understanding of the target audience and support the creation of personas and scenarios; but while it is intended to support design solutions, it generally does not explicitly contain a design solution.

Perhaps expectations for the use of probe data are best summarized by Gaver, who stated, “Most of the time the relationships between probes and proposals are more complex and difficult to trace. Our design ideas are formed from a combination of conceptual interests, technological possibilities, imaginary scenarios and ideas for how to implement them. The probes are one influence in all this.”

Probes are perhaps best implemented in industry as a technique to supplement standard industry design processes such as initial analysis, feasibility, and early design work. To ensure that the focus of the project is maintained and the effort is contained, designers should take care to plan the materials in the probe pack, set the data collection time, and predetermine the type of data examination or analysis. Probes force the designer to think about the context and motivations of use, thus building a stronger understanding of the target audience and potentially leading to better and more informed designs.    

Looking Closely at e-Learning: Vision Research Reveals Ways to Improve Children’s Experiences

photo of children with a laptop

I’m fascinated by the educational opportunities e-learning is bringing into our homes and classrooms. I’m thrilled when I see how engaged and motivated my own five-year-old daughter is when she’s working on a computer designed for kids. But, as a parent, I’ve had my concerns, too. Will staring closely at the computer screen harm her eyes? Is it better for children to read on paper than on screen? E-learning is still relatively new; when I started asking these questions in 2009, there hadn’t yet been any direct studies on computer use and the impact on children’s vision.

Fortunately, I’m in a position to find out the answers. I specialize in the perceptual aspects of the computing experience, including hearing, touching, and seeing (not tasting or smelling—yet). As a senior human factors engineer in one of Intel’s user experience research groups, I conduct research on Intel® Learning Series (ILS) computers, which are designed especially for children. I was able to initiate a series of studies with Pacific University in Oregon to look at children’s performance and comfort when using computers. Together, we were on a mission to answer key research questions to improve the overall e-learning experience for children, starting with their visual experience.

Does Screen Viewing Impact Children’s Vision?

We began our research by identifying the visual demands placed on children while working on computers. We called on the experts and asked, “Does screen viewing impact children’s vision?” First on our list was Jeffrey Anshel, OD, president of the Ocular Nutrition Society and an authority on the topic. He answered, “The good news is that there isn’t any proof that computer use causes long-term eye health problems.”

Dr. Anshel went on to explain that the potential for visual strain from computer screen viewing is often a matter of short-term discomfort, not long-term detriment to eye health. “Of course, if a person focuses close up on anything—a computer screen, TV, even a book printed on paper—for 20-30 minutes or longer without taking a visual break, that person may experience some visual strain,” he added. “The symptoms could be eye discomfort, fatigue, blurred vision, and headaches. There’s even something called Computer Vision Syndrome (CVS), which is a compilation of symptoms that arise from extended viewing, technically when the sustained demands of the task exceed the abilities of the viewer.”

I discussed these issues with my research colleague, James Sheedy, OD, Ph.D., the director of the Vision Performance Institute at Pacific University and a leading specialist on vision and computer displays. He pointed out, “We saw a lot more visual strain symptoms with the old CRT displays. The CRTs flicker, contributing to eye strain, although now you can usually increase the refresh rate on those screens, so it’s less of an issue. The CRTs are also prone to reflections, which decrease contrast and legibility. Plus, the CRTs are bulky in size, so it’s harder to place them on a surface where they’ll allow for a safe viewing distance or provide the proper viewing angle. But the high-quality displays of the LCD screens don’t flicker at all, so they’re much easier on the eyes. They’re thin and easy to position, and they tend to have anti-glare screens, at least as an option.”

Most of the newer computer screens today, and all the screens used in my company’s classroom PCs, are LCDs. But I was determined to zero in on any possible factors contributing to children’s visual discomfort and strain, even with LCD screens.

photo of children with laptops

I turned to the research my team had previously conducted, in particular our observations that children use computers quite differently than adults. An adult will typically sit down at a computer and automatically adjust for less-than-ideal conditions (for example, raising the chair, brightening the screen a bit, or putting on reading glasses). A child, on the other hand, sits down at a computer and ignores any signs of discomfort or circumstances that could result in visual strain. As Dr. Sheedy pointed out, “Children are wonderfully adaptable. But unfortunately, they also tolerate poor viewing conditions like screen glare or inadequate lighting in the classroom. And they sure don’t care about proper posture when they’re using the computer!”

I agreed, “Not to mention they don’t think to rest their eyes like adults do. Or even blink!” “That’s right. And even if something is uncomfortable, most children will simply avoid doing it,” Dr. Sheedy said. “Besides, children aren’t aware of their own vision limitations, like an adult would be, whether it’s because of poor user conditions or their own vision impairment.”

I came away with an idea my team could pursue immediately—incorporating vision-screening software into computers. That way, teachers and parents could quickly test a child’s visual acuity and be able to determine whether the child’s environment needed to be adjusted or if the child had a vision impairment that needed correcting.

Studying Children and e-Reading

Based on these early conversations with vision experts, research questions were honed and a research plan on the effects of e-reading on children’s vision relative to the factors surrounding children’s all-around comfort when using computers—visual and otherwise—was developed. The team then embarked on a series of studies in collaboration with Dr. Sheedy’s team of vision experts at Pacific University, led by Shun-nan Yang, Ph.D. Our research looked at several aspects of children’s visual comfort and reading performance, including studies on the effects of font size and display quality, factoring in screen luminance and contrast. We also incorporated studies to determine whether frequency of breaks and reading posture interact with viewing distance and visual discomfort when e-reading on handheld devices.

Our first studies, on children’s comfort and performance while reading using LCDs compared to reading on paper, were designed specifically for second to eighth grade children. We recruited fifty children and screened our young participants for visual acuity and reading comprehension skills. We then carefully customized all study instructions and selected reading materials to be age appropriate. The computer and paper displays were set in a fixed position to standardize the visual angle of the display and to set the starting visual distance. Chair and table heights were adjustable and made ergonomically correct for each individual.

During the studies we measured reading performance (word recognition, speed, and comprehension) as well as viewing distances and visual comfort. Visual comfort was gauged using both a self-reporting rating scale and an objective electromyography (EMG), which measured the muscle “squint” activity around the eye.

We learned that the novice readers (the younger children) had consistently closer viewing distances, regardless of their visual acuity. Dr. Yang articulated our hypotheses, “These readers may be inherently poorer at visual encoding because they are just learning how to read, so they need a closer viewing distance regardless of font size or display quality. Their closer viewing distances may also be the result of the visuo-ocular response they habitually adopt; they lack the ability to strategically adjust their viewing distance in relation to font size and display condition.” We all agreed that more studies were needed to test these hypotheses.

With the experienced readers (the older children) in the group, only those with poor visual acuity adopted a shorter viewing distance. When the display quality was less than ideal, it made their viewing distances even shorter. Future studies would likely reveal additional visual factors surrounding text display that could result in a closer viewing distance. We did observe that the more experienced readers appeared to be able to adjust their viewing distances in relation to text and display changes, so they may benefit the most from display quality improvements. It became clear that correcting their vision impairments would allow them to adopt longer viewing distances, which would help alleviate the minimal, but notable, visual strain and discomfort they experienced.

Our findings highlight the interplay of individual visual capacity and display quality when determining developmental readers’ visual comfort and performance. After careful analysis, we were happy to report that children with good vision could read as well on LCD computer displays as on paper, without notable differences in viewing distances and the resultant visual discomfort.

I was relieved to learn that e-reading doesn’t appear to cause additional visual strain for children with adequate vision. Any visual discomfort that was reported was very minor. But there is always room for improvement. We took copious notes on factors to research further for the development of new screen technologies.

Exploring Other UX Factors

The outcomes of our first e-reading vision studies on children with 20/20 vision were interesting. We learned that when screen luminance and contrast are adjusted properly, children can comfortably view the screen from farther away. We had also learned about variable lighting and other conditions that make classrooms unique compared to home and office environments. This influenced the latest features being developed for our classroom PCs, like sensors that could automatically detect light levels surrounding the computer and adjust accordingly.

We continued to brainstorm new computer screen design specs and setting options based on our findings. The plan was to fast track one idea for our classmate PCs—creating software that would provide pop-up reminders every twenty minutes or so to remind kids to take visual breaks and to focus on something far away, ideally twenty feet away, for twenty seconds.

In addition, a longitudinal e-reading pilot program was in the works. Short-term usability results can be so different when children are involved. Children love the e-learning experience because of technology’s “wow” factor, so they may not report long-term discomfort or other effects. Therefore, we were preparing for a pilot study in the classroom, comparing reading on computers versus reading on paper. The goal was to follow the same students from second through sixth grade to gain a long-term perspective on the effects of e-reading on children.

photo of children with a tablet

Improving Children’s e-Learning Experiences

I’m confident that the growing—and inevitable—integration of e-learning into the classroom will be a positive, exciting development. We’re off to a good start with the vision and other research we’re conducting with our partners, but there are many more factors to be studied and improved upon. Research on the effects of children’s long-term computer use is still extremely limited; there remain key research questions that need to be answered by all of us.

Computer industry influencers and other companies leveraging UX designers and developers need to continue to set new standards for the industry. We must invest in e-reading studies, make recommendations, and initiate improvements. All of us have to be proactive about sharing our research findings. As user experience professionals, we need to step forward and do all we can to advocate for children and education.Rina Doherty 是 Intel 用户体验组的一位资深人因工程学工程师,她对儿童的视觉与计算机屏幕浏览进行开拓性的研究。作为用户体验研究人员和尽责的家长,她重视与儿童浏览计算机屏幕的时间增加相关的视觉舒适性问题,而将在线学习的好处放在次要地位。

文章全文为英文版上級人間工学エンジニアでIntelのユーザエクスペリエンスグループの一員であるリナ・ドーティ(Rina Doherty)が、子どもの視覚とコンピュータスクリーンの使用に関する先駆的調査を紹介している。 ユーザエクスペリエンスの研究者、そして問題に関心を寄せる親として、彼女は、子どものコンピュータモニターの使用増加に関連した視覚的な快適さの問題と、eラーニングのメリットとを比較検討している。

原文は英語だけになります

COMMUNICATION GAP: Designing an Interface for Chinese Migrant Workers

While there are many problems to be solved in China, we chose to address the need for communication between migrant workers and their children left behind at home. Although these workers lack exposure to computer-based tools, they welcome changes to better their lives and thus, they make willing subjects. Their weaknesses—including a low rate of literacy—amplify their detachment from Western design issues and make them interesting subjects.

These workers’ migration has raised social concerns in China. There are 114 million Chinese migrating to urban centers to find work. Many of these migrant workers are parents who leave their children at home in the villages in the care of the children’s grandparents. From interviews, they confided that their main problem was parenting their children from afar. They believed that better means of communication with their children would help alleviate this problem.

As modernization of China proceeds, few efforts are specifically focused on the migrant worker population. The goal of this prototype is to improve the quality of communication between a migrant worker parent and his or her children while they are separated.

Background

During Phase 1 (April 2005–June 2005), we chose a scenario, interviewed stakeholders, and started to sketch a solution. During Phase 2 (June 2005–October 2005), we continued to identify specific design issues and started conducting user studies.

Previous Relevant Work

Chinese cultural traits that inform user interface design have been researched widely, and the number of studies grows every year as China gains in importance to Western technology companies. These studies, such as the ones documented in Geert Hofstede’s book Cultures and Organizations: Software of the Mind, are largely ethnographic, anthropological, or psychological. Less focus has been on user studies with interactive devices. The inaccessibility of the target users is most likely the reason for this.

As many parties are affected and interested in the phenomenon of urban migration (including local and national governments, local construction companies, and telephone companies, for example), there has been a great deal of sociological and economic research on the migrant worker population in China. These studies are largely aimed at understanding reasons for migration, consequences, economic factors, and policies that deal with this phenomenon. However, design considerations for communication between migrants and their families back home have not been studied from an interaction design perspective.

Research Findings

An initial interview was conducted with a migrant worker in her thirties who works as a domestic helper in a private home in Beijing. She has two children aged eleven and fourteen, whom she has left back in her village with her in-laws. Her husband is also a migrant worker in Beijing. He lives and works on a construction site.

Remote Parenting: Our interviewee said she spent 90 percent of her phone conversations with her children and only 10 percent with her parents. She made it clear that parenting her children was her primary concern. Therefore, in villages, the primary users of our communication system will be children and young adults.

Synchronization: To contact her children, our interviewee needs to call one day ahead of the regular market day and ask the shopkeeper in charge of the public phones to inform the children to wait near the public phones at a certain hour on the market day when she plans to call back. Unpredictable schedules often reduce communication between parents and children to one call per month. Minimal parental presence in the children’s lives as a result of such infrequent communication makes parenting difficult.

Literacy: Migrant workers have varying levels of literacy. They are not always familiar with Pinyin, the Romanization of the Chinese script, which is generally used for input on Chinese desktop computers.

Familiarity of Surroundings: Live-in housekeepers rarely explore the area surrounding their workplace. However, they do know where the local food market is.

Surveys indicate that China has 39,000 towns in total. Each town is frequented by inhabitants of a dozen surrounding villages for services such as its market, its public telephones, and such. We estimate that each town’s public telephone serves as the rendezvous point of communication for the children of about one hundred families. We estimate about 500 parents share the public phones located near local produce markets.

Challenge

  • Allows for asynchronous communication.
  • Is affordable by migrant workers given their economic conditions.
  • Is usable given migrant workers’ various levels of literacy and lack of exposure to technology.
  • Is accessible for migrant workers unfamiliar with their surroundings.
  • Scales up to the sub-population of Chinese migrant workers and to China’s vast geography.

Solution

The most helpful research conducted took advantage of our geographical location. The design process was informed by personal experiences of the group, two stakeholder interviews, and focused user studies.

As the research team is based in Beijing, shared personal experiences from all the researchers helped to increase the general collective knowledge on the migrant worker phenomenon, lifestyles of housekeepers in urban centers, Chinese telecommunication standards, rural phone use, and so on. As it is difficult to gain much of this information during the course of rapid prototyping (since villages can take days to reach), our collective experience proved valuable.

In addition to the interview with a housekeeper, a field visit was paid to an elementary school for migrant workers’ children on the outskirts of Beijing. The school’s dean was interviewed at length. After the interviews were performed, several designs for kiosks were proposed, and focused user studies were performed with female migrant workers who work as custodial staff in the research facility.

storyboard explaining how to use a telecommunications device

Design and Resulting Design Decisions

In order to make communication more accessible and affordable to migrant workers, we decided to design a system of public, shared voice mail kiosks at markets in towns and cities. These kiosks let parents leave messages for their children, and vice versa, in an asynchronous manner. We also augmented the voice mail with videos so as to enrich the communication between parents and children, thus increasing parental presence in villages where the children reside. With video, parents can also visually inspect their children and perhaps even their school work, while—we hope—feeling better connected.

By locating the kiosks in local food markets, the system can piggyback onto public newspaper reading displays, already prevalent in China. These displays are well-understood by the population as the source for localized, public information. Rural towns also use the same newspaper display system.

A parent wishing to communicate with his or her children must initiate the first contact by using our system to call home and leave a voice-mail message. To do this, the parent enters the public phone number of the home village (or of the nearest town with public phones) as well as his or her full name. This combination identifies the recipient(s) of the first message, namely the parent’s family at home, and serves as the rendezvous point of communication between the two parties for all future messages.

The combination of the home public telephone number and the parent’s full name grants access to the voice mailbox of that family. No user name or password is required. Such departure from the conventional account-access paradigm sacrifices privacy for simplicity. This design choice is justified by our observations that the Chinese are accustomed to trading privacy for other benefits, such as cost or simplicity of use.

Compared to entering the phone number, entering a Chinese name is a more difficult design challenge. Pinyin is not widely recognized by migrant workers, so Pinyin-based input methods that are otherwise used pervasively on desktop computers cannot be used in this case. We turned to hand-writing recognition as the next most promising technology. Our user studies indicated that a tablet PC interface might be suitable for Chinese name input.

Management of voice mailboxes in our system is simplified by not providing the feature for deleting messages. Instead, each message is stored for a fixed time interval. This design choice also prevents one’s messages from being deleted by someone else. This is crucial for such a system as ours in which accounts are not protected by passwords.

The most unique portions of this work are in its context: the evaluation of technology in the context of this specific user segment, the social context of the family and urban settings, and the leveraging of Chinese cultural cues. The usability studies validated some claims; our next steps included participatory design.

series of photographs doumenting stages of the test
Entrance to a typical produce market where migrants use the telephone. 2. A typical public newspaper display where various people come for their news. 3. On task A1, users hesitated little when asked to write their name and hometown with a stylus on the standard tablet PC interface. 4. On Task B1, fourteen of twenty users could not easily find the location of their hometown on a three by four foot map of China. A list was preferred.

User Studies

Three small usability tests were performed on each of the twenty participants to validate assumptions about literacy, readability, and input. All participants were female migrant workers who held housekeeping jobs. The participants were not employed in a private residence (as our intended users are), but in a place of business. One factor that may have affected the results was that all the participants

were accustomed to working among computers and other technology devices. They may not have been as intimidated by technology as a domestic worker in a residential scenario would be. None, however, had any experience using a tablet-style PC and stylus.

The first task, A1, was intended to see how a stylus and tablet could best support Chinese character input for this user segment (see illustrations). There is a common Chinese practice of writing ideographs in the air with a fingertip. This technique is used when describing characters to others or remembering the number of strokes in an ideograph. We initially explored the idea of letting users write their names with their fingertips in the air. The resulting strokes would be analyzed by a camera or transferred onto a tablet. But as there was a question about technical feasibility, user study B1 was administered with users entering characters with a stylus. For this task, a commercial tablet PC was used. Participants were asked to write their name in Chinese characters using the stylus. We predicted that participants would be able to enter Chinese characters on the electronic system easily after some initial hesitation. The study revealed that most participants had no problems using the stylus.

Another point in administering task A1 is related to the fact that all the participants were not hesitant to use the device. We acknowledge that this could be because a person who is seen as professionally superior to them was requesting that they perform this task.

We also acknowledge that, in our test, character recognition proved to be less than 50 percent accurate. This may have an effect on future designs.

The next issue was for a user to glance at the display to check the kiosk for new messages. One design displayed the names of hometowns that have recently sent messages atop the kiosk. Another design displayed a map of China with geographical indicators of the origins of new messages. Two tests were performed to verify if users could identify the town nearest to their hometown that had a public phone. In the first task, B1, participants were asked to identify the location of their village on a map of China. In the second task, B2, participants were asked to pick out their hometown on a list of towns. Observations from administering task B1 concluded that fourteen of twenty subjects did not easily find their hometown on a map of China. They relied heavily on text descriptors on the map. However, on task B2, all users were able to quickly find their hometown on a list. Therefore, our design eschews spatial knowledge of hometowns and uses a list format instead.

illustrations of these phone in context
1. A standalone version of the kiosk. The list of rural towns at night,grouped by province, lets users quickly glance at the kiosk to see whether or not they have new messages. 2. Close-up sketch of kiosk with stylus. 3. The kiosk sketched into a typical public newspaper display. Smaller versions of these displays are commonly located near markets.

Next Steps

The immediate next step for this study is to create and test a high-fidelity prototype. First, a kiosk would be installed at a market. Replies to user messages would be stock messages from the system. This would simplify implementation and focus on usability issues. Later, rural kiosks would be installed and the interface would be tested with children.

As the original aim was to uncover usability and design issues by designing for a real scenario, we present issues that will be studied in detail as we move forward with the project.

Issues for Further Research

Scale: A huge population with relatively small number of unique names;extensive settlement networks ranging from very poor to very rich; many local dialects.

Chinese character input: What is the best interface for Chinese characterinput and error correction?

Shared access: How do Chinese privacy and sharing norms apply to communal devices? We have observed that Chinese people have fewer privacy concerns than Westerners. How can this idea of privacy inform design?

Conceptual understanding of data: Can we design interfaces for users toconceptually understand virtual messages? Concepts would include performing operations (save, delete), navigating, keeping mental lists, and understanding the implications.

Designing for Chinese novices: How is designing for Chinese novices different from designing for novices in general? How can we communicate the usefulness of a product before it is actually used?

Literacy: How do designs for character-only readers differ from onesdesigned for users who know Pinyin or even for those who are illiterate? Does the process of designing icons for Chinese culture differ from designing icons for a Western audience since the Chinese lan-guage is derived from pictograms?

Socioeconomic extremes: How should designs for the same culture differ bysocioeconomic level? What are the personas for these levels?

Social hierarchy: What is the best way to design user studies and interfaces without social status affecting outcomes?

In our study, one of the two administrators was a foreigner, and both were full-time employees. The subjects seemed to cooperate without reluctance. We attribute this behavior to the social hierarchy inherent in the Chinese culture: one always submits to the command of one’s superior. One helpful measure was to hold conversation and interviews in a private setting (such as a home or empty hallway). For testing on the “high-tech” tablet PC, it helped to keep explanation of the device (and its implied monetary worth) to a minimum, instructing users to “just write on it as they would a piece of paper.” Participants were not paid or given rewards for their time, and there didn’t seem to be an expectation for such.

There are many possible avenues of exploration related to designing for emerging markets, specifically in China. This paper is part of a larger effort to gain practical and specific insight into the usability of various products in China. The design and implementation of this project has already, and will continue to, uncover specific interaction design experiments that can be carried out with Chinese participants.

Remote User Research: Opportunities for Adoption in Asia

User researchers are an inquisitive lot, no matter in which part of the world they work. However, there is diversity in the research discipline in different parts of the world. The practice of user research in Asia has grown significantly in the past few years. Large organizations, tech SMEs and startup unicorns are aggressively recruiting UX talent, increasingly in short supply. “Why should I hire you?” today sounds like an obsolete question when hiring for UX roles.

Researchers and project teams in Asia are experiencing higher than ever stakeholder buy-in for conducting user research. The recent shift to “all things remote” behavior has led all UXers to make an involuntary shift into gathering feedback remotely. Cloud-based collaborative design and user testing platforms such as Figma, Miro, Usertesting.com, UXArmy, dscout, etc. have become the UX work essentials.

However, despite the COVID-19 work-from-home model, usage of remote research tools and platforms in Asia remains limited. This is reflected in a survey of around 100 researchers and designers in Asia conducted by UXArmy. 25% of respondents reported research experience ranging between 8 to 20 years, while over 40% had research experience ranging between 3 to 8 years. The remaining reported less than 3 years’ experience.

Sixty percent of all survey respondents agreed with the statement that “Remote UXR is not practiced in Asia as much, in comparison to the Western countries.” This article will discuss current user research practices in Asia and make the case for incorporating more remote research as standard practice.

Figure 1
Researchers in Asia self-report conducting less remote research compared to those in the West. Alt text for accessibility: Graph. 60% of survey respondents agreed with the statement “Remote UXR is not practiced in Asia as much, in comparison to the Western countries,” compared to 29% who agreed with the statement “Remote UXR is equally practiced in Asia and the Western countries.”

FIg 1. Researchers in Asia self-report conducting less remote research compared to those in the West

Current practice of user research in Asia

The majority of user research leaders in Asia come predominantly from UX Design and market research backgrounds. Few leaders have transitioned from non-UX roles like market research, analytics, product owners, project managers, etc. The responsibilities for these roles naturally required a need to make sense of user feedback and take action on it. Most people in these roles had close interaction with end user behavior in the product development process. Prior to 2010, most companies in Asia did not have a dedicated user researcher role. Instead, UX designers doubled up as researchers when required. After 2010, with the prolific adoption of digitalization in the industry, companies wanted to de-risk their investments resulting in higher demand for specific user researchers.

At junior and entry levels, a near majority of user researchers are likely to have a formal education in research-related fields or have experience doing usability testing. Newer roles at mid-senior level in the industry are now demanding user researchers to have formal education in a field related to user research (psychology, sociology, anthropology, etc.), and sound knowledge of statistics.

While budgets allocated to user research have been increasing, they remain small in comparison to design budgets. A meager 4% of those who responded to the survey said their organization’s user research budgets exceed 10% of the product development budgets. User research budgets are typically a part of Product development or Design budgets but may also come from marketing or be a separate ad hoc allocation.

Figure 2
User Research budgets are seldom dedicated; most are taken from the Product development or Design budgets. Alt text for accessibility: Graph.81% of respondents indicated research budgets were taken from product development or design budgets, while only 13.5% stated user research had its own dedicated budget in their experience.

FIg 2. User Research budgets are seldom dedicated; most are taken from the Product development or Design budgets

A good majority of the UX Research happening in Asia is design-focused, pre-launch usability testing, or competitive benchmarking. Much of evaluative user research is planned to align with development Sprints. User validation of early-stage concepts is gradually increasing in frequency. Due to less tangible outcomes and no proven method of calculating ROI, the application of longer-term generative user research to identify latent user needs, understand user behavior or ideate new product opportunities remains low, or in some organizations, even non-existent.

Figure 3

Fig 3. Most User research in Asia happens during the product design phase

Only a handful of organizations like banks, start-up “unicorns,” and global behemoths like Google or Meta (Facebook) are investing in long-term user research in Asia. The researchers’ effort-spend chart supports the observational indication that less than 5-8% of the total UX research efforts in Asia are spent on long-term generative user research. To clarify any discrepancy, the “Understanding / Ideation” measure in the graph above includes the generative research that happens as part of the Discovery phase in the product development lifecycle.

Figure 4

Fig 4. Most of the user research in Asia is still evaluative and executed in-person

Current usage of remote user research platforms in Asia

While the remote user research platform market is hot worldwide, in Asia, usage is highly fragmented. Preferences for the range of tools available vary depending on user role and desired output. While some tools have been created for and in-market, the usage of remote research platforms remains miniscule in countries like Mainland China, Japan and Korea.

Many researchers in Asia are attracted to online research platforms that produce a quick collection of usability metrics along with graphs like Heatmaps. However, the benefit of usability metrics alone seems to pale in comparison to the platforms that provide video clips of participants using the product and sharing feedback.

Researchers focused on usability testing appreciate metrics-driven platforms, while qualitative researchers tend to prefer platforms supporting screen and video recording. For most countries in Asia, especially Southeast Asia, users are not comfortable with face recording during an Unmoderated Test. For remote user interviews, turning on the Webcam is considered normal, especially when participants are offered substantial incentives.

Despite the possibilities of remote platforms, as discussed, usage is still not high across Asia. Only 30% of researchers agreed with the statement: “For organizations to improve User intimacy, Remote UX Research benefits in a big way.” Compare this to the large selection of a neutral answer which could be heavily biased due the new normal set by the coronavirus pandemic.

This suggests remote research is not a preferred mode of user research in Asia, as a majority of respondents are not sold on remote research platforms as being robust or providing enough value from an output standpoint.

Figure 5

Fig 5. Remote User research is not seen to play a significant role in improving User intimacy

A significant percentage of user researchers in Asia also report low awareness about their options for remote platforms as a reason why they are not choosing these tools as an option.

Figure 6

Fig 6. Lack of awareness and comfort level in using traditional methods are the top two reasons to avoid using of Remote UXR platforms

The following is a list of platforms reported by the participants of the survey. This list is in no specific order and no recommendation of any specific platform is being made.

UserZoom

Used by large enterprise customers in Singapore, Malaysia.

usertesting.com

Used by large enterprise customers, mainly in Singapore.

UXArmy

Used by a mix of large enterprise customers, Small and Medium Enterprises (SMEs), and startups in Singapore, Malaysia, Indonesia, Thailand and Australia.

Lookback

Used by large enterprise customers, mainly for remote user interviews as an alternative to ZOOM, Microsoft Teams, Google Meet, etc.

Maze Design

Used primarily by  designers and design researchers in SMEs and tech startups. Popular s in Malaysia and Indonesia.

Optimal workshop

Used for Information Architecture research by large enterprise customers The Card Sorting tool is hugely popular across all of Asia.

Userlytics

Reported as the most rarely used tool in Asia.

Figure 7

Fig 7. Popularity of select Remote UXR platforms tools in Asia

 

 

The case for using remote user research in Asia

To clear the air, here is a disclaimer. I do not recommend a complete switch to remote user research. Ethnographic research in real-life environments remains one of the most effective and least biased research methods. Irrespective of user interviews, diary studies, field studies or any other research methods used, the “DO“ / “DOES” part in the Empathize phase of Design thinking is critical to be able to identify and define the problem to be solved.

However, remote user research methodologies have a role to play as the demand for research grows in Asia and teams  .

The following highlights some compelling reasons for project teams, UX designers, and researchers in Asia to include remote user research in their product development lifecycle.

#1 Research Recruitment is a top challenge

Asia is the most diverse continent.  The variations in geographical landscape, weather, culture, language, and income levels present a huge business opportunity. The eleven time zones in Asia create a tremendous challenge to schedule user research interviews in multi-country projects. A popular joke in my research circle is: “To confuse a researcher, get them to schedule user interviews with participants from Bangkok, Ho Chi Minh City, Kuala Lumpur, Jakarta and Bali.” Those cities are in four different time zones and include participants speaking four to five distinct languages.

Another difficulty in finding research participants for products with a target market across Southeast Asia is the varying communication language. In Thailand, Vietnam, Indonesia, and Myanmar the majority of people have a very strong preference to write and speak in their local language. While English is popular in the Philippines and Malaysia, it is common that people speak with an abrupt mix of local language and English.

Illustration of a few of the several language scripts in Asia

Fig. 8 Illustration of a few of the several language scripts in Asia

Unlike the Euro in the Eurozone, there is no common currency in Asia. Asia has over 40 different currencies, with each country having their own. With so much diversity, research recruitment and scheduling of user interviews is best handled if automated. This makes a clear case for automated recruitment platforms which provide end-to-end recruitment services including screening, scheduling and distribution of incentives. This is a gap that remains to be addressed by technology and remote research platform creators.

#2 Growth of research teams calls for a remote research platform

Aligned with digitization and digital transformation initiatives, many businesses in Asia have been aggressively building in-house research teams. The outsourcing of research to UX consultancies has notably reduced. The UX consultancies are still engaged to augment the in-house UX research teams to secure unbiased feedback avoiding “Déformation professionnelle”. The planning of most user research studies typically starts 3 weeks ahead. However, projects with business backing can start with little notice due to the push from stakeholders for some quick decision making. On the other hand, less user-centric organizations require months for internal convincing and decision making to proceed with projects involving user research.

Team sizes in large organizations and tech startups vary vastly. Sometimes the ratio of UX personnel to Engineer ratio in tech startups can be higher than those in large organizations. Personally, I have seen and heard of team sizes in large organizations ranging between 5 to 20 user researchers. Researchers, designers, copywriters and research ops members comprising a UX team can exceed 100 members in a handful of organizations.

With UX teams getting larger and new user researchers joining the workforce, productivity has taken the spotlight. ROI of user research attracts a greater scrutiny from the top management.

Since the time and budgets are never infinite, UX teams cannot grow indefinitely to keep up with the required scale and demand coming. The management and UX leaders have recognized this limitation, and therefore the remote research platforms are being adopted more readily.

While many user researchers may be more comfortable doing in-depth interviews and online surveys, the timeline to deliver insights from usability testing during development needs to align with development Sprints. The agile model of development hardly leaves a time margin to properly plan an execution of research using the traditional user research methods. As businesses in Asia are no exception to this reality, either the research is selectively conducted for some Sprints or research teams have resorted to using a cloud-based user research platform.

 

#3 Need gaps in existing Remote UX Research platforms

Top need gaps reported by researchers who are already using remote tools include:

  • Access to participants matching screening requirements; more variation in behavioral attributes of participants than offered in the existing marketing user panels
  • Availability of platforms offering automated participant recruitment
  • Integration with existing design tools, online repositories for research, and (client-provided) user panels
  • Accommodation of technical limitations of participants. e.g. poor internet connection, lower tech savviness, fear of installing malicious software
  • Localization in existing platforms, e.g. language translations
  • Setting clear expectation on information received from remote unmoderated testing; depth of participant responses from unmoderated studies is not comparable to those from moderated interviews

Conclusion

By nature, Asia is fast moving. Being ahead of the pack to win titles and rewards is admired in Asian culture. Creating a great user experience demands a longer-term outlook as the ROI of user research is not always immediately visible. That is one of the potential reasons why the practice of user research had a delayed start in Asia. In some countries like Japan and Korea, technology teams still drive UX decision making.

Research in Asia is changing fast with rapid digitalization. More and more prominent businesses from the West are setting up offices in Asia to benefit from a “not to be missed” opportunity. Emerging tech unicorns like Grab, Carsome, Ninjavan, Carro, Traveloka, VNG, Lineman wongnai, CRED, Zerodha, are making headlines. Large organizations based out of Asia have also stepped up their design budgets. They are setting up user research teams in the quest of creating products that are highly relevant to the target audience and deliver a superior user experience.

Researchers in Asia have high expectations from the remote research platforms and are currently underserved. The variety in language, geographies, and user attributes like literacy gaps and income presents a huge challenge to the creators of remote platforms looking to market to the researcher community in Asia. With challenge comes the opportunity, and the companies building remote research platforms stand a good chance to be among the next wave of startup unicorns.

Book Review: Of Testing and Techniques

Beyond the Usability Lab: Conducting Large - Scale Online User Experience StudiesA review of
Beyond the Usability Lab: Conducting Large-Scale Online User Experience Studies
By Bill Albert, Tom Tullis, Donna Tedesco.
Morgan Kaufmann, 2010

In 2008, UXMagazine devoted an issue to remote usability testing guest edited by Tomer Sharon. As managing editor at the time, I confess to having had a rather skeptical view of the value of remote testing.

One of the articles that made me re-think my attitude was a case study comparing the relative usability of two sets of information about the Apollo program, one provided by NASA, and one by Wikipedia. That case study was written by Tom Tullis, one of the authors of Beyond the Usability Lab, which takes the themes introduced in that issue (Vol 7, Issue 3, 2008) and explores them in detail.

Traditional usability testing has been focused on small sets of users, and the book does not take issue with this approach. Indeed Bill Albert said to me in an interview,  “…if the goal is really just to identify usability issues, I think I’d fall in line with a lot of other people, saying six to eight users is plenty to identify the significant usability issues with a particular design.” (User Experience podcast, episode 55, www.uxpod.com).

What the 2010 book concerns is the fact that web technology “enables us to move beyond the lab and efficiently conduct user experience studies with a much larger sample of users.”

The book begins with an introduction describing what the authors mean by “online usability studies,” including a description of when such studies are appropriate, what one can expect to achieve, and the strengths and limitations. Remote studies are good for comparing designs, for collecting detailed and scalable usability metrics, and for exploring design issues in the users’ own environment, with all its attendant complexities. On the other hand, there are many instances (such as identifying the major usability issues with early prototypes) when other methods are preferred.

The book is well-structured for the practitioner. After the introduction, the following three chapters explore planning, designing, piloting, and launching the study. The hands-on approach is reminiscent of Rubin’s (and now Chisnell’s) classic Handbook of Usability Testing, in that it contains sufficient detail to enable a practitioner to engage the method with a degree of confidence.

Chapters 5 and 6 discuss data preparation, analysis, and presentation.  Chapter 7 provides good in-depth analysis of specific tools (Loop11, RelevantView, UserZoom, and WebEffective) that can be used to conduct remote studies, as well as advice on choosing the appropriate tool for your own study. Chapter 8 discusses discount methods, including “totally homegrown techniques for the adventurous.”

Chapter 9 presents seven case studies of remote research conducted with between 24-300 users with a range of tools.

Throughout the book, specific examples illustrate concepts and methods. The authors provide detailed instructions for using Microsoft Excel to calculate appropriate averages and confidence intervals. There is also advice on dealing with data gathered from open-ended questions (when simple numerical analysis is not adequate).

The authors describe how to identify and deal with data from “flat-liners”—participants who complete studies as quickly as possible to obtain the associated incentive.

It’s a real pleasure to encounter a book that not only takes the reader on a journey through the rich possibilities of technique, but does so in a manner that is clear, readable, and accessible. I was particularly pleased with the simple explanations of statistical techniques, which are so often presented as incomprehensible.

If you’re interested in any of the following questions, you can look to this book for practical and effective answers:

  • Should you conduct a between-subjects or within-subject study?
  • What variables do you need to consider?
  • How can you deal with outliers?
  • How can you calculate and display confidence intervals?

The book does not shy away from the difficulties involved in conducting remote research. For example, if you want click-stream data, it may be necessary to have participants install or allow a plug-in, which may mean you can’t test with so-called novice users.

If I were to complain, it would be about the need for a chapter specifically on conducting studies on mobile devices—an area that is ripe for a similarly detailed “how-to” guide.

Whether you’ve conducted remote studies in the past and want to extend your capability and knowledge, or you are a complete newcomer, this excellent book is a necessary companion on your journey from the lab into the world outside. You will refer to it often, and it will alert you to opportunities and dangers. What more could you ask of a book?