The Problem with Snapshot Research

I recently completed a study on an agentic AI-powered, personalized news aggregator. My findings showed that, although users celebrated the tool’s diversity and editorial independence, 3 months later the AI adjusted its model based on user engagement patterns, filtering out the unique perspectives that participants originally praised. This highlights a critical methodological problem: Most UX research assumes the system remains the same, but agentic AI behavior is a moving target, rendering snapshot findings obsolete as the system adapts.

Agentic AI systems disprove the assumption of a system behaving predictably between releases. Agentic AI systems develop a unique working relationship with each user that’s shaped by the user’s input history. The quarterly forecast an AI agent produces today may bear little resemblance to the forecast it produces 6 months from now, after it has absorbed how its user actually works.

The responsiveness to the real-world way the system is used is the value proposition of agentic AI; the unique benefit of an agentic system is its increasing utility over time as it learns the user’s context, preferences, and edge cases. But for those of us who study these systems, it creates a problem that our standard methods cannot handle. An experience that was tested last quarter is not the experience the user is interacting with today.

The challenge becomes acute when the most important UX outcome is trust. Trust is inherently longitudinal over time. A snapshot study capturing the moment alone describes almost nothing about how the relationship is changing, which misses the evolving relationship arc.

We need methods that track the relationship as it develops over time. However, we don’t need to invent new methods. We can adapt approaches our field already uses, combining them with frameworks from psychology based on decades of human relationship studies.

The following sections present three approaches that I have used or am actively developing in my own work on enterprise AI. None of the methods is finished, but all, I think, indicate the direction in which our research practice needs to progress.

Modified Diary Studies for Evolving AI

In a traditional diary study, a participant records what they did and how they felt, but for agentic AI, a researcher must also track what the AI was doing in parallel. The agentic AI behavior may have changed in ways the user noticed but did not articulate. Structuring entries around these two parallel timelines allows researchers to see when user sentiment and AI behavior diverge or converge, providing analytical value that a single-track diary would miss.

Trigger Entries on Deviation as Well as the Calendar

Diary studies typically use fixed intervals. Participants log an entry every day or every few days on a schedule. For agentic AI, the most useful moments are often the unexpected ones. When the AI does something surprising, positively or negatively, the user’s reaction in that moment is more informative than any scheduled reflection.

I have started building event-triggered entries into my study designs alongside regular interval entries. The instructions are simple: If the AI does something unexpected, take a moment to log it. The bar for what counts as unexpected is intentionally low so we capture the small moments of surprise as well as the dramatic ones. Trust is calibrated through accumulated small signals as well as the occasional big ones.

Capture What the AI Was Actually Doing

This modification requires collaboration with engineering. To understand the evolving user experience, data on the AI’s evolving output is necessary. During the study, record what the AI was recommending or generating for the user at regular intervals, ensuring that appropriate privacy controls are in place. Without behavioral data from the system, a researcher can only form an impression without evidence.

A Concrete Logging Schema for AI Output

Partnering with engineering to capture system data provides the necessary context for qualitative findings. When setting up a longitudinal study, request a logging schema that captures the following specific data points at the exact time of the user’s diary entry.

• Raw output: The exact artifacts the AI presented to the user.

In a news aggregator study, this means logging the specific headlines and article summaries displayed on the user’s dashboard.

• Output distribution: The categorical breakdown of the recommendations.

A log might show the feed contained 60% local news, 30% national politics, and 10% international finance.

• Confidence levels: The system’s internal probability score for its recommendations.

Low confidence scores frequently align with moments in which a user reports a drop in predictability.

• Feature weights: The underlying variables driving the output.

Did the system prioritize the user’s recent click history, explicitly stated preferences, or globally trending topics?

Reviewing this log alongside a participant’s interview transcript reveals exactly why a system’s perceived benevolence or predictability changes over time.

Study Duration

A practical note on study duration: For meaningful AI adaptation to occur, researchers typically need a minimum of 4-6 weeks. Anything shorter and the study is likely covering a relatively static version of the system. For deeper questions about trust development and skill formation, 8-12 weeks is more useful. Longer studies require careful attention to avoid participant fatigue: Use lighter entries, flexible cadences, and clear incentives. I have found that small, weekly check-ins beat lengthy daily diaries in studies that run longer than a month.

Diary studies allow us to observe the day-to-day user experience, but they are not enough to understand the deeper sentiment of trust. To truly track that relationship, we must also measure trust as a multidimensional construct.

Longitudinal Trust Measurement

Research consistently shows that trust is a multidimensional construct. Hancock et al.’s 2023 meta-analysis of over 300 published studies found that trust, a multidimensional construct, is often broken down into core components like competence, benevolence, and predictability. This framework, traced by Sodern and Hertel, is often applied to human organizations and workplace relationships. It provides a useful lens for evaluating user-agent interactions. Understanding these dimensions is essential as they develop at different rates and are impacted by different user-agent interactions.

Four Dimensions of Trust

To better understand the nuances of this relationship, we can break down trust into four distinct components. Each of these parts develops in its own way and provides us with a clearer view of how a user’s confidence evolves over time.

Competence trust evaluates whether the AI performs the task well. This dimension calibrates quickly; users build or lose competence trust based on observed outcomes and adjust their assessment with each interaction. A few successful recommendations from the AI build competence trust. An obvious judgment error can reset it. This is the dimension most teams already measure, even if they call it accuracy, satisfaction, or something similar.

Benevolence trust concerns whether the AI is working in the user’s interest. This dimension is the slowest to develop and, once damaged, the hardest to repair.

Predictability trust is the user’s understanding of the AI’s rationale. A lack of understanding creates an uncomfortable dependency, as the user relies on a system they cannot explain or anticipate. Predictability trust builds slowly and breaks easily.

Data groundedness evaluates the degree to which a user can verify the information the AI provides. An agent might consistently generate accurate answers, building competence trust. If the system cannot show the user exactly which documents or datasets informed those answers, trust erodes rapidly. Groundedness requires the AI to cite sources, provide audit trails, and anchor its outputs in verifiable facts. For example, a logistics manager reviewing a supply chain recommendation would want to see the specific inventory reports driving the system’s logic.

Measuring Trust as a Moving Target

Once researchers treat trust as four dimensions rather than one, the measurement approach changes. A practical structure includes brief pulse surveys at regular intervals, supplemented by periodic semi-structured interviews. The pulse surveys produce a quantitative signal that can be charted over time. The interviews show the reasoning behind the numbers.

Pulse surveys should capture confidence in accuracy, understanding of AI rationale, perceived alignment with the user’s goals, and the ease of verifying source data.

The cadence of measurements matters. I’ve found weekly pulses for the first month, then biweekly thereafter, have worked well in my studies. Less frequent pulses might miss rapid early calibration, but if it’s more frequent, participants start to give rote answers lacking depth or nuance.

The most useful data point in trust research is the change in score before and after a specific event. These before-and-after readings show the dynamics of trust repair and erosion in a way no single snapshot ever could.

Measuring trust as a single score hides important details. In this example, a 12-week longitudinal view of user trust in an agentic AI system, perceived benevolence and data groundedness remain flat, yet competence steadily climbs as the system learns the user’s daily routines. At week 4, an unexplained background update causes predictability to plummet. Tracking these dimensions separately reveals exactly how silent system updates damage a user’s mental model, even while the tool’s actual competence continues to grow.

Figure 1. Tracking the dimensions of trust over time (generated with Google™ Gemini™).

Measuring trust dimensions provides a detailed view of confidence, but we can also take a broader perspective and examine the entire user relationship. By viewing the interaction as a developing story with predictable stages, we can better understand the arc of collaboration. The following method maps how this journey unfolds over time.

Relationship Arc Research

This method borrows conceptually from Knapp, Vangelisti, and Caughlin’s interpersonal relationship research in psychology. The premise is that a user’s relationship with an AI agent follows a predictable arc that we can study in stages. If the study recruits participants at the point of first exposure to an agentic system and then follows the participants through months, the study can map how that relationship develops, where it gets stuck, and what causes it to deepen or fall apart.

Four Stages of a Relationship

In adapting Knapp’s relationship framework for UX research, I focus on four distinct stages of the relationship users develop with AI tools. This distinction allows targeting questions and honing an understanding of the point at which a user exists on this relationship progression.

Encounter is the first stage. The user has just encountered the AI. What expectations do they bring, and what mental model forms based on the initial interactions?

Exploration is the second stage. The user repeatedly engages with the AI, probing its capabilities and limitations. They notice when the AI matches or surprises their expectations. Work-arounds and early habits of reliance or skepticism develop.

Calibration is the third stage and the most consequential. The user develops a working model of the AI’s actual capabilities and limitations. If calibration goes poorly, the user either trusts too much (which is ruined) or trusts too little (and abandons a useful tool).

Establishment or breakdown is the final stage. The user has settled into a stable relationship, either integrating the AI productively into their work or disengaging from it. Studying this stage reveals what successful long-term human-AI collaboration looks like and what its failure modes are.

Designing the Research Schedule

The practical difference from a calendar-based longitudinal study is that the interview schedule aligns with stage transitions, not arbitrary intervals. Interviews are not needed at weeks 1, 4, 8, and 12. Interview at the encounter, when the user enters exploration, when calibration is taking shape, and when the relationship has either been established or broken down.

A longitudinal study requires developing indicators for stage transitions. Some are behavioral, such as the user starting to make conscious decisions about what to delegate. Some are linguistic, in which users transition from evaluative third-person terms to collaborative first-person terms or to passive, defeated language.

Different users move through the stages at different rates: Some users accelerate, some get stuck, and some loop backward. For instance, a system update that changes the user’s expectation of AI behavior can push an established user back into exploration.

Researchers track these behavioral markers through daily diary entries and user interviews to pinpoint a participant’s exact stage in the relationship. Table 1 outlines the four primary stages of human-AI collaboration alongside the specific linguistic cues users typically express.

Table 1. Relationship Stages in Human-AI Interaction and Corresponding Linguistic Indicators

Relationship Stage	User Mindset	Behavioral and Linguistic Indicators
Encounter	Forming initial expectations	Uses third-person, evaluative language
Exploration	Probing capabilities and limitations	Developing early work-arounds and habits
Calibration	Establishing a working model of the AI	Making conscious delegation decisions; uses first-person collaborative terms
Establishment / Breakdown	Integrating the tool productively, or disengaging	Productive collaboration, or defeated language

Making It Work in Practice

Start with snapshot research but plan for longitudinal studies. Running a traditional usability study or concept test for an agentic AI product is useful for initial validation. The critical change is building longitudinal touchpoints into the research plan from the start. Even a lightweight monthly pulse survey, layered on top of snapshot work, provides a view of how the system’s experience is evolving. The monthly pulse can be an early warning system for the moments when the snapshot findings might start to drift from the lived user experience.

Working with engineering to log the AI’s behavior changes at intervals that align with the research schedule will transform what can be analyzed. Work to correlate shifts in user experience with shifts in system output. Without this, the results are guesswork.

Present findings as trajectories, not static numbers. Charting trust scores over time shifts the conversation from “Does it work?” to “How is the relationship developing?” This is the question that truly matters for agentic products.

Looking Ahead

UX research on products featuring AI is at an inflection point. The methods most of us were trained on were built for static systems in which the artifact supporting the study is static long enough to evaluate. Agentic AI does not work that way, and the gap between the systems we are now researching and the methods we are using to study them keeps growing.

The methods described in this article are starting points that will evolve as agentic AI research matures. We must extend our existing skills: listening, observing, and capturing experience in systems that demand a longer view.

Quick Start: Your First Longitudinal AI Study

• Log the AI, not solely the user: Partner with engineering to capture system outputs alongside user diaries.
• Trigger entries via surprise: Ask users to log an entry the moment the system does something unexpected.
• Measure the four dimensions of trust: Track competence, predictability, benevolence, and data groundedness independently.
• Map the relationship stages: Conduct interviews when users transition between the stages of encounter, exploration, calibration, and establishment.

References

Amershi, Saleema, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, et al. 2019. “Guidelines for Human-AI Interaction.” Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems: 1–13. https://doi.org/10.1145/3290605.3300233

Hancock, P. A., Theresa T. Kessler, Alexandra D. Kaplan, Kimberly Stowers, J. Christopher Brill, Deborah R. Billings, Kristin E. Schaefer, and James L. Szalma. 2023. “How and Why Humans Trust: A Meta-Analysis and Elaborated Model.” Frontiers in Psychology 14: 1081086. https://doi.org/10.3389/fpsyg.2023.1081086

Knapp, Mark L., Anita L. Vangelisti, and John P. Caughlin. 2014. Interpersonal Communication and Human Relationships. 7th ed. Boston: Pearson.

Lee, John D., and Katrina A. See. 2004. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors 46 (1): 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

Sondern, Dominik, and Guido Hertel. 2024. “Revisiting the Classic ABI Model of Trustworthiness: Interactive Effects of Trustworthiness Components on Trust in Mixed-Motive Social Exchange Contexts.” Journal of Trust Research 14 (2): 213–236. https://doi.org/10.1080/21515581.2024.2388659

Yang, Qian, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. “Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design.” Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems: 1–13. https://doi.org/10.1145/3313831.3376301